AI search fix
GPTBot user-agent: what it is and how to allow it
GPTBot is the HTTP user-agent string OpenAI uses when its crawler fetches public web pages for ChatGPT-related retrieval and indexing workflows. You will see it in server logs, CDN dashboards, and tools like Microsoft Clarity Bot Activity. It is not the same as ChatGPT-User or OAI-SearchBot, which may appear for other product flows. To welcome GPTBot, allow it in robots.txt and ensure your CDN or WAF does not return 403 or bot challenges to that user-agent. Allowing GPTBot does not guarantee ChatGPT will cite or recommend your site.
When auditing AI-readiness, search logs for the exact string GPTBot (case-sensitive in many tools). If counts are zero while Googlebot is healthy, check Cloudflare Bot Fight Mode and robots.txt before rewriting marketing copy.
GPTBot quick reference
- Operator: OpenAI — crawler for ChatGPT-class retrieval.
- Typical log string: GPTBot (verify current documentation from OpenAI).
- Not the same as: ChatGPT-User, OAI-SearchBot — review each UA separately.
- Policy layers: robots.txt Allow + CDN/WAF skip rules for verified AI crawlers.
- Verify: server or Cloudflare logs, Clarity Bot Activity, or Express Check on your domain.
You'll get an HTML report listing whether GPTBot is allowed in robots.txt and at your edge.
Run Express CheckRelated questions
Updated