Are AI crawlers like GPTBot blocked by default?
No at the robots.txt layer and increasingly yes at the infrastructure layer, and that split answer is the whole story. Robots.txt works on permission-by-absence: no file or no matching rule means GPTBot, ClaudeBot, and PerplexityBot may crawl. But your CDN and firewall don’t ask robots.txt, and Cloudflare has blocked known AI crawlers by default on every new domain since July 1, 2025. In our scan of the web’s top 1,500 domains, 14.2% blocked AI user agents at that infrastructure layer, and most of the blocking looked accidental.
What is the default in robots.txt?
Allowed. The robots.txt standard only restricts; it never needs to grant. A site with no robots.txt, or a robots.txt that never mentions AI bots, is fully open to all of them. That is why 82% of the top sites we scanned still permit GPTBot: doing nothing permits it.
The trap is inheritance. A blanket User-agent: * / Disallow: / rule, often left over from a staging site or an old anti-scraper tutorial, binds every crawler that lacks its own named section, AI crawlers included. Your file can block GPTBot without containing the string “GPTBot” anywhere. Our robots.txt guide has a checker that tests all five major crawlers against your live file in seconds.
What does Cloudflare block by default?
Since July 1, 2025, every new domain onboarded to Cloudflare blocks known AI crawlers unless the owner opts out, and existing customers got a one-click version of the same block a year earlier. Cloudflare sits in front of a huge share of the web, so this single default quietly flipped the answer to this page’s question for millions of sites.
If your site went onto Cloudflare after mid-2025 and nobody visited the Bots settings since, assume you are blocking AI crawlers right now, whatever your robots.txt says.
Which other defaults block AI crawlers?
Bot-protection presets that never mention AI. WAF managed rules and “block bad bots” toggles at most CDNs and security plugins bundle AI crawlers into the same category as scrapers, and challenge screens do the rest: an AI crawler served a “verify you are human” page simply leaves.
None of this is visible from your browser, which is what makes it the most damaging failure our scanner finds. The firewall fix guide has an embedded tester plus the exact settings to change in Cloudflare and the major WAFs.
How common is accidental blocking?
Common enough to assume it until tested. In our top-1,500-domain study, 22.1% of scanned sites block at least one major AI crawler in robots.txt and 14.2% refuse AI user agents at the firewall.
Some of that is deliberate policy, and the giants blocking on purpose skew those numbers, but the accidental version has a signature we saw constantly: robots.txt says welcome while the firewall says no, or a wildcard rule from 2019 blocks bots its author never heard of.
How do you find out what your site is doing?
Test both layers, because they fail independently. Read your robots.txt for named AI-crawler rules and for wildcard disallows, then test what your infrastructure actually serves to an AI user agent, which no file can tell you.
The free scan does both in one pass: it checks all five crawlers against your robots.txt and fetches your page with an AI-crawler user agent to catch the firewall layer, then links the fix guide for anything that fails.
See where your site stands. The free scan takes about fifteen seconds and shows every fix.
Run a free AI visibility scan