AI search fix
GPTBot user-agent: what it is and how to allow it
GPTBot is the HTTP user-agent OpenAI sends when its crawler requests public pages for ChatGPT-related retrieval and indexing. You will spot it in origin logs, CDN bot dashboards and Microsoft Clarity Bot Activity. It differs from ChatGPT-User and OAI-SearchBot, which may appear for other product flows. To permit GPTBot, add an explicit Allow in robots.txt and confirm your WAF or CDN does not return 403 responses or bot challenges to that string. Allowing GPTBot is a crawl prerequisite only — it does not guarantee ChatGPT will cite or recommend your site.
When reviewing AI readiness, filter logs for the exact token GPTBot — many tools treat it as case-sensitive. If counts stay at zero whilst Googlebot crawls normally, inspect Cloudflare Bot Fight Mode and your live robots.txt before rewriting page copy.
GPTBot quick reference
- Operator: OpenAI — crawler for ChatGPT-class retrieval.
- Typical log string: GPTBot (confirm against current OpenAI publisher guidance).
- Not the same as: ChatGPT-User, OAI-SearchBot — set policy for each string.
- Policy layers: robots.txt Allow plus CDN or WAF skip rules for verified AI crawlers.
- Verify: origin or Cloudflare logs, Clarity Bot Activity, or Express Check on your domain.
You'll get an HTML report listing whether GPTBot is allowed in robots.txt and at your edge.
Run the diagnosticRelated questions
- GPTBot blocked by Cloudflare — how to unblock itEdge WAF and Bot Fight Mode fixes when GPTBot never reaches origin.
- robots.txt for AI crawlers — how to write itExplicit Allow sections for GPTBot and peer AI user-agents.
- ChatGPT cannot crawl my website — how to fix itEnd-to-end crawl checks beyond the user-agent string alone.
Updated