Write a robots.txt that welcomes AI crawlers
Your scan failed because your site has no readable robots.txt. Crawlers will still visit, but you’re leaving their behavior to defaults, and you’re missing the one place where you can explicitly welcome AI bots and point everything at your sitemap. The fix is a small text file and about ten minutes.
What should your robots.txt say?
For most businesses that want AI visibility, this is the whole file:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlThe named groups are technically redundant next to User-agent: *, and worth keeping anyway. They document intent, and they protect those bots if someone later adds a blanket disallow to the wildcard group. Each token maps to a real system: GPTBot feeds ChatGPT, ClaudeBot feeds Claude, Google-Extended governs Gemini’s use of your content. Our guide to how AI crawlers work covers what each one does with what it reads.
What mistakes actually block AI bots?
Three patterns account for nearly every failure we see. A blanket Disallow: / under User-agent: *, usually left over from a staging site, blocks everything that lacks its own named group. Copied “block AI scrapers” files from 2023 tutorials explicitly disallow the exact bots you now want.
And wildcard rules like Disallow: /*? intended for faceted-search noise can catch far more than intended, because in robots.txt the longest matching rule wins and * matches anything. When in doubt, test a rule mentally against the URL /: if any disallow pattern matches it and no longer allow rule does, that bot is out.
Where does the file go on your platform?
It must be reachable at yoursite.com/robots.txt, and every platform has its own door:
- WordPress: use your SEO plugin’s editor (Yoast: Tools → File editor; Rank Math: General Settings → Edit robots.txt). Plugins generate a virtual file, so there is nothing to upload.
- Shopify: edit the
robots.txt.liquidtemplate (Online Store → Themes → Edit code → Templates). Shopify’s default already allows most crawlers; add the named AI groups above the default rules. - Webflow: Site settings → SEO → robots.txt, paste the file contents, publish.
- Static hosts and frameworks: drop
robots.txtin your public folder so it deploys to the web root. - Cloudflare: note that Cloudflare offers a managed robots.txt feature that can prepend AI-blocking directives. If your file keeps showing rules you didn’t write, that setting is why.
Does robots.txt actually stop or allow anything?
It’s a published preference that reputable crawlers honor, and all the major AI crawlers state they do. But it controls nothing at the network level, which cuts both ways. A bad actor can ignore it, and, more importantly for you, your firewall can override it: since mid-2025 Cloudflare blocks AI crawlers by default on new domains regardless of what your robots.txt says.
That’s why our scanner tests both layers separately. If robots.txt passes and the bot-fetch check fails, the problem lives in your firewall, not this file.
How do you verify the fix?
Load yoursite.com/robots.txt in a browser and read it as a crawler would, top to bottom. Confirm the file loads as plain text rather than an error page or your site’s 404 template, confirm the Sitemap line points at a sitemap that actually resolves, check the page itself carries no noindex or noai directives, and then rescan.
The robots.txt check and all five per-bot checks re-evaluate on every scan, so you’ll see exactly which rules each bot resolves against.
See where your site stands. The free scan takes about fifteen seconds and shows every fix.
Run a free AI visibility scan