How AI crawlers read your site
AI crawlers are bots that fetch web content to feed AI systems, some for training data and some for live answers. Five of them decide most of your AI visibility, each obeys or ignores your rules in its own way, and two separate layers of your infrastructure decide whether they get in at all. Block the wrong one at either layer and you vanish from the product it feeds. This guide covers who they are, how they decide what to crawl, and where sites accidentally lock them out.
Which AI crawlers matter?
These five cover the major AI answer surfaces:
| Crawler | Operator | What it feeds |
|---|---|---|
| GPTBot | OpenAI | ChatGPT’s models and search |
| ClaudeBot | Anthropic | Claude’s models |
| PerplexityBot | Perplexity | Perplexity’s answer engine |
| Google-Extended | Gemini training (not Google Search rankings) | |
| CCBot | Common Crawl | The open dataset many AI labs train on |
One nuance worth knowing: Google-Extended is not a separate crawler at all. Googlebot does the fetching, and Google-Extended is a control token in robots.txt that tells Google whether the fetched content may feed Gemini. Disallowing it doesn’t touch your search rankings, which is why so many sites toggled it off in 2023 and forgot.
How do they decide what to crawl?
The same way classic crawlers do. They request /robots.txt, look for a group addressed to their user-agent token, fall back to the * group when they lack one, and follow the allow and disallow rules with longest-match precedence.
That inheritance rule is where most accidents happen: a blanket Disallow: / under User-agent: *, added years ago against scrapers, silently binds every AI crawler that lacks its own named group. Our robots.txt guide has a copy-paste file that welcomes all five explicitly, which also protects them from any future blanket rules.
What’s the second gate most sites forget?
The firewall, and it outranks robots.txt. Your robots.txt is a published preference; your CDN enforces actual access, and the two frequently disagree. Cloudflare has blocked known AI crawlers by default on every new domain since July 1, 2025, with a one-click block available to everyone else, and WAF bot lists bundle AI crawlers with scrapers.
A site can pass every robots.txt check and still serve every AI bot a challenge screen. This is exactly why our scanner fetches your page twice, once normally and once with an AI-crawler user agent, and compares what comes back. When the two differ, the firewall guide shows where the block lives.
Do training crawls and live retrieval differ?
Yes, and the difference shapes what blocking costs you. Training crawls, which is GPTBot in its training role plus CCBot and Google-Extended, determine what models know long-term. Live retrieval, which is PerplexityBot and ChatGPT’s search fetching pages at answer time, determines what can be cited today with a link a user might click.
Blocking training bots limits your presence in model knowledge, a slow and diffuse cost. Blocking retrieval bots removes you from cited answers immediately, which is the fast and visible cost. Ahrefs’ 1.4M-prompt study found the overwhelming share of ChatGPT citations flow through ordinary search retrieval, so the retrieval path is where most businesses have the most to lose.
Should you ever block AI crawlers?
Sometimes, and it should be a decision rather than a default you inherited. Publishers whose text is the product have a coherent case for withholding training data, which is why the blocking tools exist. For most businesses the calculus runs the other way: you want AI systems to know you exist, describe you accurately, and cite you when your customers ask.
Blocking the crawlers while hoping to appear in AI answers is a contradiction, and the accidental version of it, via an old robots.txt or a CDN default, is the single most damaging misconfiguration we find.
How do you check your own site?
Run the free scan. It tests all five crawlers individually against your live robots.txt, fetches your page with an AI-crawler user agent to catch firewall blocks, and measures your response speed, since retrieval systems skip slow sources. Every failure links to its fix guide. For the wider picture of what these crawlers feed and why it matters commercially, start with what AIO means.
See where your site stands. The free scan takes about fifteen seconds and shows every fix.
Test all five crawlers on your site