AI search fix
AI crawler user-agents: GPTBot, ClaudeBot, PerplexityBot and more
AI crawler user-agents are the HTTP User-Agent strings that identify which company's bot is fetching your pages — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended and Google-Other for Google, Applebot-Extended for Apple, and Meta-ExternalAgent for Meta. Each is a separate policy decision in robots.txt and at your CDN or WAF — allowing one does not allow the others, and a broad security template can silently block all of them at once. None of these strings guarantee AI citations; they only control whether the bot can reach your pages at all.
Treat this as a reference table, not a one-time checklist: new user-agents appear as vendors add products, so re-check your robots.txt against current documentation periodically rather than assuming a policy written months ago still matches today's bot list.
AI crawler user-agents at a glance
| User-agent | Operator and typical role |
|---|---|
| GPTBot | OpenAI — crawler for ChatGPT-class retrieval and training-data collection. |
| OAI-SearchBot | OpenAI — separate search-oriented fetch flows, not the same policy as GPTBot. |
| ChatGPT-User | OpenAI — often tied to a live, user-initiated browsing action inside a chat. |
| PerplexityBot | Perplexity — crawler for answer and citation pipelines. |
| ClaudeBot | Anthropic — crawler for Claude search and citation-oriented retrieval. |
| Google-Extended | Google — generative AI and model-training crawl, separate from Googlebot. |
| Google-Other | Google — non-search fetch category; review requested paths before blocking. |
| Applebot-Extended | Apple's crawler policy string — manage separately from generic Applebot. |
| Meta-ExternalAgent | Meta — crawler identifier; verify current documentation before policy. |
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report listing which AI crawler user-agents are allowed or blocked in robots.txt and at your edge.
How to identify each bot in your logs
The User-Agent string is the only reliable signal — do not infer bot identity from IP ranges alone unless you are matching against a vendor's published list, since ranges change. Log the full User-Agent header on your key pages and group by exact string, not a loose "contains GPT" filter, which can blend several bots into one misleading bucket. A healthy crawl shows repeated GET requests returning HTTP 200 or 304; 403, 503 or endless challenge pages usually mean a WAF or CDN block, not a robots.txt problem.
OpenAI runs three separate crawlers
GPTBot, OAI-SearchBot and ChatGPT-User are three distinct strings, and a robots.txt block written for one does not apply to the others — allowing GPTBot does not automatically allow search-oriented OpenAI traffic unless OAI-SearchBot gets its own explicit line. ChatGPT-User is also different in kind: OpenAI notes it can reflect a live user action inside a chat session, so robots.txt rules may not apply the same way as to a scheduled crawl. Write one dedicated User-agent block per string rather than assuming a single rule covers "all OpenAI traffic".
Google-Extended and Google-Other are not Googlebot
Googlebot handles classic Search indexing — rankings, Discover, and much of AI Overviews sourcing. Google-Extended is a separate user-agent for generative AI and model-related crawls, including Gemini-oriented access; blocking it does not stop Googlebot from crawling for normal search, and allowing Googlebot does not automatically allow Google-Extended. Google-Other covers fetch activity outside classic Search crawling — review which paths it requests before deciding policy, rather than treating it as a synonym for Googlebot.
Smaller and newer crawlers: Applebot-Extended, Meta-ExternalAgent
Applebot-Extended and Meta-ExternalAgent are less-documented policy strings than GPTBot or Googlebot, but the same discipline applies: confirm the string appears in your logs and note which URLs it requests, decide an explicit allow, limit or block policy based on your own requirements, then align robots.txt and any CDN bot-management rules with that decision. Avoid a default block just because a string is unfamiliar — an observation period before acting gives you real evidence instead of a guess.
Setting an allow policy: robots.txt and the CDN edge
Every bot above needs the same two policy layers, and both must agree. First, robots.txt: add one User-agent block per bot with an explicit Allow for the paths you want crawled — a catch-all Disallow under User-agent: * silently overrides every narrower allow. Second, your CDN or WAF: Cloudflare Bot Fight Mode and similar managed rules bundle AI crawlers into generic bot-management heuristics by default, so add a skip rule per user-agent string, or GPTBot-class bots can receive a 403 before your origin is ever reached.
Verify the policy worked with evidence, not assumption: filter server or CDN logs for each exact user-agent string and confirm repeated HTTP 200 or 304 responses on public pages, not 403 or 503. If edge and origin logs disagree, fix the CDN rule before touching robots.txt again.
Frequently asked questions
Do all AI crawlers respect robots.txt?
Most major AI crawlers generally follow robots.txt directives on sites that publish them, but a WAF or CDN rule can still block a request before robots.txt is ever read.
Is ChatGPT-User the same as GPTBot?
No. GPTBot is a scheduled retrieval crawler; ChatGPT-User can reflect a live, user-initiated action inside a chat session. Policy them separately in robots.txt.
Should I block a crawler user-agent I do not recognize?
Not automatically. Confirm the requested paths and check current vendor documentation first — an unfamiliar string is not by itself evidence of a problem.
Does allowing these bots guarantee AI citations?
No. Allowing a crawler only removes a technical access block. Whether a platform cites or recommends your site depends on content quality and its own policies.
How is PerplexityBot different from Perplexity-User?
PerplexityBot is automated crawling for retrieval and citation pipelines. Perplexity-User typically reflects a live person's session — browser or app traffic — not the scheduled crawler. Treat the two strings as separate policy decisions if both appear in your logs.
Should I allow ClaudeBot and anthropic-ai?
They can serve different flows. ClaudeBot is the primary crawler for page fetching; anthropic-ai and the older Claude-Web string may appear separately in logs. Check current Anthropic documentation before allowing additional strings, and give each one its own robots.txt section rather than assuming a single rule covers all of them.
Related questions
- robots.txt for AI crawlers — how to write itTemplate for trusted AI user-agent sections.
- Server rules for AI bots: Cloudflare, Nginx and how to verify access in logsCloudflare WAF/Bot Fight Mode and Nginx allow rules, plus log verification.
- AI crawlers blocked — how to fix GPTBot, PerplexityBot, ClaudeBot and Google-ExtendedDiagnose robots.txt and CDN/WAF blocks per bot, plus the Google-Extended policy decision.
Updated