How do I know if I'm blocking AI crawlers?

Check your robots.txt for Disallow rules under AI user-agents like GPTBot, ClaudeBot, PerplexityBot, or Google-Extended. This tool reads your robots.txt and tells you, per crawler, whether it's allowed or blocked.

Should I allow AI crawlers?

If you want AI answer engines like ChatGPT, Claude, Perplexity, and Google's AI Overviews to describe and cite your brand, they have to be able to read your site. Blocking their crawlers makes you invisible in AI answers.

AI crawler check - is your site blocking GPTBot, ClaudeBot?

AI crawler check

If your robots.txt blocks the AI crawlers, the answer engines can't read or cite you - full stop. Enter your site to see, per crawler, whether GPTBot, ClaudeBot, PerplexityBot and the rest are allowed in.

We read your /robots.txt only · nothing is stored

Why this matters

AI answer engines build their view of your brand by crawling your site. Many sites copied old robots.txt rules - or pasted a "block all bots" snippet - that quietly shut out the AI crawlers. The result: you go invisible in ChatGPT, Claude, Perplexity, and Google's AI Overviews, and you never see it happen.

Allowing the crawlers is necessary but not sufficient. Once they can read you, grade your overall AI-readiness - and to see what the engines actually say, start a trial scan.