Free AI SEO tool
Choose how recognized AI crawlers should be handled, then copy a syntactically simple robots.txt block or append it to pasted rules.
Allow is selected by default and emits an explicit Allow rule. Choose Don't specify to leave a crawler out of the generated block.
Collects public web content that may help improve future OpenAI models.
Indexes content for ChatGPT search results and citations.
Crawls public content for Anthropic's model and product work.
Indexes pages to surface and link them in Perplexity search results.
Fetches a page in response to a user's request in Perplexity.
Controls Gemini training and grounding use without affecting Google Search ranking.
Builds Common Crawl's public web dataset, used in research and model development.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: Amazonbot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Diffbot Allow: / User-agent: Timpibot Allow: /
Useful context for applying this tool to your site.
Usually no. A crawler is normally allowed when it has no matching disallow rule. Explicit Allow rules can document an intentional policy or override a broader setup, depending on the crawler's implementation.
No. Crawl permission does not guarantee indexing, citation, training use, or a mention in an AI answer. It only removes one possible access restriction for crawlers that honor robots.txt.
Yes. Paste your existing content into the optional merge field. The generated AI crawler rules are placed after a clear comment separator so you can review the final text before publishing.
No. robots.txt is a crawling preference, not access control. Use authentication and authorization to protect pages that must remain private.