Unblock ClaudeBot so Claude can see your site
Your robots.txt blocks ClaudeBot, the crawler Anthropic uses to gather web content for Claude. Claude powers a fast-growing share of AI conversations - plus assistants and agents built on its API - and right now none of them can read your site.
Where is the block coming from?
Look for one of two patterns in your robots.txt: an explicit User-agent: ClaudeBot group with a Disallow: /, or a catch-all User-agent: * disallow that applies because ClaudeBot has no named group. Anti-scraper lists that circulated in 2023–2024 included ClaudeBot by default, so many sites block it without ever having decided to.
What’s the fix?
Add an explicit allow group (or delete the blocking rule):
User-agent: ClaudeBot
Allow: /Because named groups override *, this works even if your general rules stay restrictive. If you also see anthropic-ai or Claude-Web entries from older lists, allow or remove those too - stale tokens are harmless when allowed, damaging when disallowed. The same copied blocklists usually catch PerplexityBot as well.
How do you verify the fix?
Deploy the updated robots.txt and run the scan again. If it still fails, your CDN or bot-protection layer may be challenging the crawler - check firewall rules for “AI” or “scraper” categories, and see the firewall check.
See where your site stands. The free scan takes about fifteen seconds and shows every fix.
Run a free AI visibility scan