The Split Test Pro Crawler
What SplitTestPro-Crawler is, why it visits your site, and how to control it — a polite, robots.txt-honoring crawler that captures up to 20 of your own pages to ground AI test plans.
Split Test Pro’s Testing Overview builds AI test plans from your real pages. To ground those plans in what your site actually shows, our crawler captures a small snapshot of the pages you test on. This page documents how it behaves.
Identification
The crawler identifies itself on every request with this User-Agent:
SplitTestPro-Crawler/1.0 (+https://splittestpro.com/documentation/crawler)When it visits
The crawler only ever visits your own site, and only when:
- you (or a teammate) click Refresh site snapshot in the Surfaces dialog on the Testing Overview page,
- you generate or build a test in a plan and a stored page snapshot is older than 7 days.
It never crawls sites that aren’t connected to your Split Test Pro workspace, and it never follows links off your domain.
How politely it crawls
- At most 20 pages per crawl run — one representative page per surface in your surface map.
- 1 request per second, at most 2 pages fetched concurrently.
- At most one crawl run per store at a time, and manual refreshes are limited to once per hour.
- robots.txt is honored: pages your robots.txt disallows are skipped. The one exception is URLs you typed into your surface map yourself — entering them is explicit consent to capture them.
- No cookies or credentials are ever sent. The crawler sees your pages the way a logged-out first-time visitor does.
What it stores
For each captured page we keep a trimmed copy of the page’s HTML — scripts, styles, and inline images are stripped, and the result is capped at 60 KB — plus a short structural summary (headings, buttons, forms). Snapshots power plan generation and one-click test builds for your workspace only, and are deleted with your workspace.
Blocking the crawler
Add this to your robots.txt to stop the crawler from visiting anything it discovers on its own:
User-agent: SplitTestPro-Crawler
Disallow: /Pages listed directly in your surface map are still captured (you added them); remove them from the Surfaces dialog to stop those captures too.
