"Who is the best family dentist in Mississauga?"
Lakeview Dental comes up most often for families. Sources highlight same-week appointments and upfront pricing. Bright Smiles Centre is also mentioned, though some reviews note long wait times.
New AIOScan now runs inside ChatGPT
AI crawler access
AI Crawler Checker
Find out which AI crawlers can reach your website and which ones you are turning away. The scanner reads your robots.txt rule for each bot, then fetches your page as an AI crawler to catch firewall blocks. Free, about fifteen seconds, no signup.
Free · takes about 15 seconds · no signup, no email required
Instant results. No signup, no email. Full report, not a teaser.
Prefer chat? Ask ChatGPT “Can AI read example.com?” with the AIOScan app.
The AI crawlers that matter
GPTBot feeds OpenAI and ChatGPT. ClaudeBot feeds Anthropic's Claude. PerplexityBot builds Perplexity's answer index. Google-Extended is the robots.txt token that controls whether Gemini may use your content, separate from normal Google Search. CCBot builds Common Crawl, the open dataset many AI models train on. Block one and that assistant knows less about you. How each AI crawler works.
Where do AI crawler blocks hide?
In three places. robots.txt, as a Disallow under a bot's name or a blanket rule under User-agent: *. Your firewall or CDN, which can answer a bot with a 403 or a challenge page even when robots.txt allows it. And meta tags such as noai, which tell crawlers not to use the page. The scan covers robots.txt and the firewall directly. Write a robots.txt that welcomes AI crawlers.
Why robots.txt alone is not enough
Most crawler checkers only parse robots.txt. That misses the most common block we see: a firewall that challenges anything announcing itself as a bot from a cloud server, which is exactly what AI crawlers do. This checker sends a real request identifying as an AI crawler and reports what came back. Fix a firewall that blocks AI crawlers.
What happens after access?
Access is the first of four groups in the AIO Score. Once the crawlers get in, the scan checks whether your content sits in the raw HTML, whether it is structured so a model can parse it, and whether it leads with answers an assistant can quote. How the AIO Score works.
The scan behind every checker runs the same open methodology. Read exactly how the AIO Score works before you trust any score, ours included. Prefer to have the fixes handled for you? See how the agency works.
Frequently asked questions
Short answers first. That happens to be the whole philosophy.
Which AI crawlers does the checker test?
GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended for Gemini, and CCBot from Common Crawl. Each is checked individually against your robots.txt, and the scanner also fetches your page identifying as an AI crawler, so a firewall or CDN block shows up even when robots.txt allows it.
How do I check if GPTBot can crawl my site?
Run the scan above. It reports whether your robots.txt allows GPTBot and whether your server actually serves the page to a request identifying as GPTBot. Both have to pass. A robots.txt that allows GPTBot behind a firewall that challenges it is still a block.
Are AI crawlers blocked by default?
Not by robots.txt, which allows everything unless told otherwise. But some CDNs and hosts block AI crawlers at the firewall by default, and Cloudflare has done so for new domains since mid-2025. That is why the scanner tests a real fetch, not just your rules.
Should I allow every AI crawler?
If you want customers to find you through AI assistants, allow the crawlers behind the answers they use: OpenAI's, Anthropic's, Perplexity's and Google's. Training-only crawlers are a judgment call, but blocking them means future models know less about your business.
Does llms.txt replace robots.txt?
No. robots.txt controls who may crawl. llms.txt is an optional guide that points AI tools at your best content. A site needs a correct robots.txt first; llms.txt is a nice extra after that.