A File You've Never Opened
Every site has a robots.txt file. Visit yourdomain.com/robots.txt right now and you'll see it.
The file tells crawlers what they can read. Most site owners never write it by hand. A CMS default, a security plugin, or an old IT ticket adds a rule meant for spam bots. That rule often blocks AI crawlers too, and nobody notices until AI referral traffic stays flat at zero.
Training Bots and Citation Bots Are Different Bots
OpenAI, Anthropic, and Perplexity each run more than one crawler, and the crawlers do different jobs. Most robots.txt guides skip this distinction.
GPTBot, ClaudeBot, and Google-Extended pull content to train future models. ChatGPT-User, Claude-User, and Perplexity-User fetch a single page live. That happens the moment a user asks something the AI can't answer from memory alone. OAI-SearchBot and Claude-SearchBot crawl to build a search index, closer to how Googlebot works today.
Most companies only mean to opt out of training. A blanket rule blocks the citation bots by accident, and that's the traffic they actually wanted.
Google Is a Special Case
Google-Extended isn't a separate crawler. It's a control token inside robots.txt.
Google already crawls your site with Googlebot for Search. Google-Extended tells Google whether that same content may also train Gemini and Vertex AI. Blocking it doesn't touch your Search ranking or your appearance in AI Overviews. It only opts out of the training use.
Check Your Site in Two Minutes
Open a browser and go to yourdomain.com/robots.txt. Look for two patterns.
First, a blanket block. This disallows every crawler, AI included, with no exceptions:
Second, a named block. This disallows one specific bot by name:
- Training bots: GPTBot, ClaudeBot, Google-Extended
- Citation bots: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User
Any of these followed by Disallow: / blocks that bot and nothing else.
The Fix
Decide what you want first. Most operators want their content quoted in AI answers. Fewer want it folded into a model's training data with no attribution.
A workable default allows the citation bots and blocks the training bots:
Prefer visibility over the training question? Drop the three Disallow blocks and allow everything.
What This Actually Costs
None of the citation tactics in our GEO fundamentals guide matter if the crawler never reaches the page. A blocked bot can't read your sources, your statistics, or your first-hand expertise. It just skips you and cites the next site on the list.
Where We're Honest About the Limits
Robots.txt is a request, not a lock. OpenAI, Anthropic, and Perplexity each publish a compliance policy stating their bots honor it. That's a claim from the company running the bot, not an independent audit.
Robots.txt also does nothing about a crawler that ignores it. That's a bot-management problem, not a GEO problem, and it needs a different tool.
Bot names change. Anthropic updated its published crawler list in February 2026. Check the source list every few months instead of copying a blocklist once and forgetting it.