Does this checker fetch my live robots.txt file?
No. This is a paste-only checker, so it does not request your domain or any URL. Paste the final file contents you want to inspect.
Free AI SEO tool
See which AI crawlers your robots.txt blocks.
Paste your robots.txt below to check AI crawler access in the browser. This tool does not fetch a URL or inspect your live site.
How to use this tool
- Paste your robots.txt content into the Paste your robots.txt field.
- Click Load example to see how a blocked crawler is reported.
- Click Check crawler access to parse the pasted rules.
- Read the blocked-crawler summary and the status beside each crawler.
- Click Show all crawlers to review the full directory of supported user-agent tokens.
Questions
Frequently asked questions
Useful context for applying this tool to your site.
What does partially blocked mean?
Partially blocked means a matching crawler group disallows at least one path but not the whole site. The checker treats Disallow: / as a full block and other disallow paths as partial access.
Does User-agent: * affect AI crawlers?
It can. When a crawler has no more specific group, many robots.txt parsers use the wildcard group as a fallback. Review broad rules alongside crawler-specific directives.
Why does an allowed result not guarantee the bot can crawl?
The result only reflects the pasted robots.txt rules. A firewall, login requirement, rate limit, DNS problem, or crawler policy can still prevent an actual request.