AI Crawlers Your SEO Platform Doesn't Check

Semrush says healthy. Googlebot reaches every page. ChatGPT still recommends your competitor.

The gap is usually AI crawlers SEO tools don't check — not content quality, not keyword rankings. This post explains one thing: why a green SEO audit can hide an AI block, and which crawlers from OpenAI, Anthropic, and Perplexity your platform never tested.

For robots.txt syntax and bot-by-bot rules, see our GPTBot guide and AI bots in robots.txt. This post stays on the SEO-platform blind spot.

SEO platforms test Googlebot. AI companies send their own crawlers.

Ahrefs, Semrush, and Screaming Frog exist to improve Google Search. Their crawlers mimic Googlebot — status codes, internal links, page speed, mobile signals.

ChatGPT, Perplexity, and Claude don't borrow Google's index for live recommendations. Each company sends its own crawlers to read the public web. Same robots.txt standard. Different user-agent names. Cloudflare, security plugins, and WAF rules may treat them differently than Googlebot.

OpenAI documents three crawlers. Anthropic's official crawler page lists ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (user-initiated fetches) — each blockable separately in robots.txt. Perplexity's publisher programme documents PerplexityBot. Google's crawler overview covers Googlebot only. Most SEO audits test none of the AI company crawlers.

For the broader GEO tools vs SEO platforms split, see the pillar. This post is the technical reason green SEO doesn't prove AI access.

OpenAI: three crawlers, one common Cloudflare mistake

GPTBot is not the ChatGPT search bot. Block GPTBot, keep OAI-SearchBot open — many sites stay visible in ChatGPT answers.

The mistake: one Cloudflare Block AI bots toggle, WAF custom rule, or WordPress plugin blocks "all AI bots." Google's Googlebot passes. OAI-SearchBot gets 403. Your Ahrefs score stays 90+.

Full user-agent strings → GPTBot guide.

Perplexity and Anthropic: separate channels

Perplexity composes answers from sources PerplexityBot can fetch — not from Google's index.

Anthropic's Claude uses Claude-SearchBot for search indexing and Claude-User for user-initiated fetches — Anthropic documents that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Blocking ClaudeBot (training) is a separate decision, like GPTBot vs OAI-SearchBot.

Your Semrush dashboard tracks Googlebot. It does not confirm whether PerplexityBot reaches your product pages or whether ClaudeBot can read your services site.

Example — two AI crawlers, one firewall:

SEO report: healthy. ChatGPT: can't read you. Perplexity: might work — but without llms.txt, citations still go to competitors with clearer structured data.

BuzzStream found 49% of top news publishers block OAI-SearchBot — and 71% block at least one AI search crawler — while Googlebot access often stays open. Business sites repeat the pattern when Cloudflare's Block AI bots managed rule is enabled: Cloudflare states it blocks GPTBot, ClaudeBot, Bytespider, and others, and that this rule takes precedence over Super Bot Fight Mode — including "allow verified bots."

If the symptom is "how much is this costing us?" see how buyers use ChatGPT to find vendors. For diagnosis buckets when the site is missing from ChatGPT, see website not showing in ChatGPT.

The green-SEO / invisible-AI pattern

1. Site ranks on Google
2. SEO platform audit scores high (tests Googlebot-like signals)
3. Cloudflare Bot Fight Mode, Shopify security app, or robots.txt blocks AI search crawlers
4. Owner hears competitor named in ChatGPT or Perplexity
5. Owner commissions content work — wrong layer

Before rewriting content or subscribing to Profound or Otterly for citation tracking, confirm crawlers can arrive. Monitoring on a blocked site charts a problem you can't fix from a dashboard.

For what to buy after you confirm the blind spot, see do you need both SEO and GEO tools.

How to check without assuming a specific product

Options owners use in practice:

1. Developer review — Check robots.txt and Cloudflare bot rules against OpenAI's published user-agents
2. Server logs — Look for 403s on OAI-SearchBot or PerplexityBot while Googlebot returns 200
3. Readiness scan — Automated check for crawler access, llms.txt, and structured data (several tools offer this; some are free)
4. Manual fetch — curl with the documented user-agent string from your terminal or ask your dev

Any of these beats assuming a green Ahrefs report covers OpenAI's search crawler.

What to do next

Key takeaways

• AI crawlers SEO tools don't check because audits test Googlebot — not OpenAI, Perplexity, or Anthropic readers.
• Cloudflare and blanket "block AI" rules often kill ChatGPT visibility while Google rankings stay green.
• Confirm crawler access before content rewrites or Profound/Otterly subscriptions.