AI search fix

AI crawlers blocked — how to fix GPTBot, PerplexityBot, ClaudeBot and Google-Extended

AI crawlers get blocked in one of two layers: robots.txt disallows the user-agent, or your CDN/WAF — most often Cloudflare — challenges or returns a 403 before robots.txt is ever read. GPTBot, PerplexityBot and ClaudeBot are usually caught by broad `Disallow: /` rules or generic bot-management defaults never meant to target trusted AI traffic. Google-Extended is different: it is a separate user-agent from Googlebot used for Gemini and AI-training crawls, so blocking it is a policy decision about AI training access, not a technical accident to fix. Diagnose both layers before concluding a bot is blocked, and treat 'ChatGPT cannot crawl my site' as two checks — edge first, then robots.txt — rather than one toggle. None of the fixes below guarantee that a platform will cite or recommend your site; they only remove a technical access barrier.

Treat this as a recurring check, not a one-time fix: security plugins, CDN templates and CMS robots editors regularly reset these rules, so re-verify after any WAF, plugin or hosting change.

AI crawler blocks at a glance

Crawler	Where it usually gets blocked
GPTBot	Cloudflare Bot Fight Mode or Managed Rules most often; a robots.txt disallow less often.
PerplexityBot	A robots.txt disallow rule or a WAF challenge — check both before assuming a fix worked.
ClaudeBot	All-bot disallow snippets and aggressive bot-management defaults, more than a deliberate block.
Google-Extended	An explicit robots.txt disallow — usually a deliberate policy choice, not an accident.
Any AI bot at once	A catch-all `Disallow: /` under `User-agent: *`, which silently overrides every narrower Allow above it.

Free · 2 minutes · no card

See what AI crawlers hit on your site

Technical blockers, missing context, weak AI-readiness signals — in one HTML report.

You'll get an HTML report showing which AI crawler user-agents are blocked in robots.txt or at your CDN/WAF edge.

Check both layers: robots.txt and the CDN/WAF edge

Start with the bots you actually want to serve — GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Bingbot — and give each one its own user-agent section in robots.txt with an explicit Allow. A single catch-all Disallow under User-agent: * overrides every narrower rule above it, which is the most common reason a site "fixes" robots.txt and stays invisible anyway. Keep a separate, stricter policy for unknown scrapers so you do not lose real anti-abuse coverage while opening the door to bots you trust.

robots.txt only tells a bot it is welcome; it does not make the request succeed. If Cloudflare or another WAF challenges the crawler before your origin ever sees it, robots.txt is irrelevant — the request never gets that far. Check Bot Fight Mode, Managed Rules and any "Block AI Bots" toggle for each user-agent, not just the generic bot-management summary.

When a fix does not seem to work, treat it as a diagnostic chain rather than a single toggle: confirm the edge allows the request, then confirm robots.txt agrees, then confirm the page itself returns stable HTML for that bot — no redirect loops, JS-only shells, or endless challenge pages.

GPTBot and Cloudflare: the most common block

Cloudflare's Bot Fight Mode and Managed Rules bundle AI crawlers into generic automation heuristics by default, so GPTBot can receive a 403 or a JS challenge with no explicit rule ever written against it. In Cloudflare Dashboard → Security → WAF, add a custom rule that matches User-Agent contains GPTBot, action Skip for Bot Fight Mode and Managed Rules, then Allow — and repeat the same pattern for PerplexityBot, ClaudeBot, Google-Extended and Bingbot rather than assuming one rule covers all of them.

Match the same policy in robots.txt with an explicit Allow block, then wait roughly 24 hours and verify with a fresh server log or a curl request using the GPTBot user-agent — a repeated HTTP 200 or 304 is the only real confirmation the block is gone.

PerplexityBot and ClaudeBot: same pattern, different string

PerplexityBot is blocked the same two ways as GPTBot — a User-agent: PerplexityBot disallow rule, a broad catch-all, or a WAF challenge — and the fix mirrors it: a dedicated robots.txt section plus a Cloudflare skip rule for that exact string. PerplexityBot is not the same as Perplexity-User, which typically reflects a live person's session rather than the scheduled crawler, so policy the two strings separately if both appear in your logs.

ClaudeBot is most often caught by an all-bot disallow snippet copied from a generic "block scrapers" template, not a deliberate decision. Give it its own User-agent: ClaudeBot section with Allow paths for public marketing and docs pages, add a matching Cloudflare skip rule, and confirm the production robots.txt — not the CMS dashboard preview — actually serves that policy. anthropic-ai and the older Claude-Web string can appear separately in logs; check current Anthropic documentation before deciding whether to allow those too.

Google-Extended: a policy decision, not a technical accident

Google-Extended is a separate user-agent from Googlebot, used for Gemini and AI-training crawls rather than classic Search indexing. Blocking it does not stop Googlebot from crawling for normal Search, and allowing Googlebot does not automatically allow Google-Extended — the two are independent lines in robots.txt. Decide allow-or-block based on your AI-training and content-licensing policy, not on a guess about ranking impact: blocking Google-Extended should not affect traditional Google Search rankings, since those crawls run through Googlebot.

Document the decision your team makes — allow both, block only Google-Extended, or defer to legal for regulated content — and re-check the live robots.txt after CMS or SEO-plugin updates, since those tools often reintroduce a blanket AI disallow without warning.

Frequently asked questions

Why does Cloudflare block GPTBot and other AI crawlers by default?

Bot Fight Mode and Managed Rules bundle AI crawlers into generic automation heuristics. Without an explicit allow or skip rule for each user-agent, they can be challenged or blocked the same way as scrapers.

Is PerplexityBot the same as Perplexity-User?

No. PerplexityBot is the automated crawler for retrieval and citation pipelines; Perplexity-User typically reflects a live person's session. Treat the two strings as separate policy decisions if both appear in your logs.

Should I allow ClaudeBot and anthropic-ai?

They can serve different flows. ClaudeBot is the primary crawler for page fetching; anthropic-ai and the older Claude-Web string may appear separately. Check current Anthropic documentation and give each string its own robots.txt section.

Is Google-Extended the same as GPTBot?

No. Google-Extended is Google's own user-agent for Gemini and AI-training crawls; GPTBot is OpenAI's crawler for ChatGPT. They are unrelated strings with independent robots.txt policy.

Does blocking Google-Extended hurt my Google Search rankings?

It should not. Traditional Google Search indexing and ranking run through Googlebot, a separate user-agent from Google-Extended.

Can I allow AI search bots and still block broad training crawlers?

Yes. robots.txt is evaluated per user-agent, so you can allow bots used for search and citation while keeping stricter rules for crawlers focused on model training.

Why is my site still invisible to AI after I fixed robots.txt?

Your CDN or WAF may still be blocking the request before robots.txt is ever read. Cloudflare or another edge firewall has to allow the same bots you declared in robots.txt.

Does unblocking a crawler guarantee it will cite or recommend my site?

No. Allowing a crawler only removes a technical access block. Whether a platform cites your pages still depends on content quality, relevance and each platform's own policies.

Updated 2026-07-29