Enter a domain and this AI crawler access checker shows, in one place, what the site’s policy files say and how its server actually responds to AI crawlers. It reads the robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and other named AI user agents, checks for llms.txt, ai.txt and RSL licensing, and then sends a live request as each crawler to compare the server’s real answer against what robots.txt promises.
Runs on our server. The lookups are sent from srvscripts.com, so the result shows what one server in one location sees. Results are cached for up to 5 minutes. Our server is not allowed to test itself, so srvscripts.com shows as "not tested".
Policy versus the live response
robots.txt is advisory: it states a preference, not a lock. A CDN, WAF or security plugin can refuse a crawler that robots.txt actually allows, and a crawler can ignore a Disallow. That is why this tool keeps two columns apart: the policy (what robots.txt says for each crawler) and the observed HTTP response (what the server returned to that crawler’s User-Agent). The case worth catching is “allowed in robots.txt but refused by the server” — a bot you meant to let in that never gets through.
Training crawlers versus answering crawlers
GPTBot, ClaudeBot, CCBot, Amazonbot and the Google-Extended and Applebot-Extended control tokens concern model training. OAI-SearchBot and PerplexityBot fetch pages to answer a question and usually link back. Many sites allow the answering crawlers and decide separately about training, so the tool reports each one on its own line rather than giving a single yes/no.
How to read the result honestly
The live test runs from srvScripts’ server using each crawler’s User-Agent string. It shows how the site treats those User-Agents from our IP — not a request from the crawler operator’s own verified IP ranges, and the “browser” row is a User-Agent string too, not a real browser running JavaScript. An HTTP 401 or 403 can be a User-Agent block, but it can also be ordinary authentication or access control; the tool reports what it observed, not the cause. Read-only: it sends GET requests to the entered domain only and never follows redirects.
Checking many domains at once? The bulk AI Crawler Access Checker runs the same checks across a whole list on Apify.