← Fix-it library

Stop your firewall from blocking AI crawlers

UpdatedJuly 18, 2026· by Abdul Jaafar

When we fetched your page identifying as an AI crawler, we didn’t get your content. We got a block or a challenge screen. This is the most damaging failure the scanner can find, and the cruelest, because it is invisible from your side: robots.txt says welcome, the firewall says no, and the firewall wins every time.

Why is this the number one silent killer?

Because it defeats everything else you might fix. Your robots.txt can allow every AI bot and your content can be perfectly structured, and none of it matters if the request meets a 403 or a “verify you are human” page before your content is served. AI retrieval systems fetch dozens of candidate pages per answer and simply move to a source they can read.

There is no retry queue and no appeals process. A blocked page is just absent from the answer.

The check that catches this works the way the problem works: we request your page a second time using an AI-crawler user agent and compare what comes back. If a normal visitor gets your page and the “bot” gets a challenge screen, something upstream is filtering by user agent.

Where does the block come from?

Almost always from your CDN or security layer rather than anything you configured deliberately. Three usual suspects:

• Cloudflare. Since July 1, 2025, Cloudflare has blocked known AI crawlers by default on new domains, and its one-click “block AI bots” setting predates that. Roughly a fifth of the web sits behind Cloudflare, so this single default has quietly removed an enormous number of sites from AI retrieval.
• WAF bot lists. AWS WAF, Akamai, Sucuri, and similar products ship curated bot categories that bundle GPTBot, ClaudeBot, and PerplexityBot in with scrapers.
• Security plugins. Wordfence and similar WordPress plugins maintain blocklists that many owners enabled years ago and never revisited.

How do you fix it in Cloudflare?

Log in to the Cloudflare dashboard and open your domain, then go to Security → Bots. You are looking for two settings. First, “Block AI bots” (labeled “AI Scrapers and Crawlers” on some plans): set it to off, or to “Block on hostnames with ads” if you want a middle ground.

Second, Super Bot Fight Mode: if “Definitely automated” is set to Block or Challenge, AI crawlers are caught in that net too, so set it to Allow or add skip rules for the bots you want.

Then verify from the outside, because dashboards lie by omission: run curl -A "GPTBot" https://yoursite.com/ from a terminal, or just rescan here. Cloudflare’s own bot documentation covers the per-plan differences.

How do you fix it in a WAF or security plugin?

The shape is the same everywhere: find the bot rules, find the category that includes AI crawlers, and carve out the ones you want.

• AWS WAF: check whether the AWSManagedRulesBotControlRuleSet is attached. Its CategoryAI and SignalNonBrowserUserAgent rules catch AI crawlers. Change the rule action from Block to Count for those categories, or add explicit allow rules for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended user agents.
• Wordfence and similar WordPress plugins: look under the blocking or firewall tab for user-agent rules. Search the block list for “GPT”, “Claude”, “Perplexity”, and “CCBot” and remove those entries.
• Server-level rules: grep your nginx or Apache config for http_user_agent conditions. Old anti-scraper snippets from tutorials frequently match AI bots.

How do you verify the fix worked?

Test from outside your network, with the user agent the crawlers actually send. Two ways that take a minute:

1. Run curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yoursite.com/ and confirm you get your HTML back rather than an error page or a challenge.
2. Rescan your site. The firewall check re-fetches your page as an AI crawler on every scan, so a pass here is a pass in the real world.

While you are at it, confirm the robots.txt layer agrees with the firewall layer. The two are independent, and passing one while failing the other is common. Our robots.txt guide covers that half, and how AI crawlers work explains what each bot feeds.

Should you ever block AI crawlers on purpose?

Sometimes, and it should be a decision rather than a default. Publishers whose text is their product have a real interest in withholding training data, which is exactly why Cloudflare built the toggle. But if you are a business that wants to be found, recommended, and cited, blocking the crawlers while hoping to appear in AI answers cannot work.

The 6 points this check carries make it the largest single retrievability check in our methodology, because in the field it is the failure that most often zeroes out everything else.