Why do AI crawler user-agent tokens matter?
A robots.txt rule only applies to the user-agent token it names. Knowing the exact token helps you make intentional rules instead of blocking or allowing broad categories by accident.
Free AI SEO reference
A practical directory of AI crawler user-agent tokens.
Understand which crawler tokens appear in robots.txt, what each operator says the crawler does, and whether robots.txt is an available opt-out method.
How to use this tool
- Find the crawler name or operator you want to understand.
- Read the crawler purpose and category beside its user-agent token.
- Check the robots.txt column before relying on a crawler rule.
- Use the opt-out method to decide whether a robots.txt directive fits your policy.
- Open the related robots.txt builder or checker when you are ready to work with your file.
Crawler names in robots.txt are exact user-agent tokens, not broad product labels. This curated reference groups common AI-related crawlers by operator and explains what a published token is intended to do. Policies change, so use the official links where present before making a high-stakes rule.
OpenAI
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Collects public web content that may help improve future OpenAI models. | Training | Yes | Disallow in robots.txtOfficial docs |
| OAI-SearchBot | OpenAI | Indexes content for ChatGPT search results and citations. | Search and retrieval | Yes | Disallow in robots.txtOfficial docs |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks it to access that page. | User-requested fetch | Yes | Disallow in robots.txtOfficial docs |
Anthropic
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| ClaudeBot | Anthropic | Crawls public content for Anthropic's model and product work. | Training | Yes | Disallow in robots.txtOfficial docs |
| Claude-User | Anthropic | Fetches public pages to complete a user-requested Claude task. | User-requested fetch | Yes | Disallow in robots.txtOfficial docs |
| Claude-SearchBot | Anthropic | Indexes content to improve search results in Claude. | Search and retrieval | Yes | Disallow in robots.txtOfficial docs |
Perplexity
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| PerplexityBot | Perplexity | Indexes pages to surface and link them in Perplexity search results. | Search and retrieval | Yes | Disallow in robots.txtOfficial docs |
| Perplexity-User | Perplexity | Fetches a page in response to a user's request in Perplexity. | User-requested fetch | Review | No public robots.txt opt-out for user-requested fetchesOfficial docs |
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding use without affecting Google Search ranking. | Training | Yes | Disallow in robots.txtOfficial docs |
Common Crawl
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| CCBot | Common Crawl | Builds Common Crawl's public web dataset, used in research and model development. | Training | Yes | Disallow in robots.txtOfficial docs |
Apple
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Applebot-Extended | Apple | Controls use of public content for Apple's generative foundation models. | Training | Yes | Disallow in robots.txtOfficial docs |
ByteDance
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Bytespider | ByteDance | Collects web content for ByteDance products and model development. | Training | Review | Disallow in robots.txt, but verify with server controls |
Amazon
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Amazonbot | Amazon | Crawls content to improve Amazon products and may support Amazon AI models. | Training | Yes | Disallow in robots.txtOfficial docs |
Meta
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Meta-ExternalAgent | Meta | Fetches public content for Meta AI and related external-agent experiences. | Training | Yes | Disallow in robots.txt |
Diffbot
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Diffbot | Diffbot | Extracts structured web data for Diffbot's knowledge graph and data products. | Training | Yes | Disallow in robots.txtOfficial docs |
Timpi
| Bot name | Operator | Purpose | Category | Robots.txt | Opt-out method |
|---|---|---|---|---|---|
| Timpibot | Timpi | Crawls public web content for Timpi's distributed search index. | Search and retrieval | Yes | Disallow in robots.txt |
Ready to apply a policy? Generate a crawler-specific block or paste your current file to see how these tokens are affected.
Questions
Frequently asked questions
Useful context for applying this tool to your site.
Is this a real-time crawler directory?
No. It is a curated reference, not a live crawl log or a promise that every operator's policy is unchanged. Check the linked official documentation before making a high-stakes policy decision.
Do all AI crawlers honor robots.txt?
No. robots.txt is a widely used convention, but compliance depends on the crawler and the type of request. The directory flags known exceptions and user-triggered fetchers separately.
Can one crawler have more than one purpose?
Yes. Operators can use related crawlers for model improvement, search retrieval, and user-requested browsing. This directory classifies each token by its primary documented use to keep the reference readable.