AI search fix
GPTBot blocked by Cloudflare — how to fix it
GPTBot is OpenAI's web crawler for ChatGPT. If Cloudflare's WAF blocks it, your pages stay invisible to ChatGPT even when they rank well on Google. By default Cloudflare's bot fight rules treat GPTBot as a threat and silently return a 403 or a JS challenge, so the answer never includes you.
The fix is two-step: explicitly allow GPTBot at the Cloudflare edge so requests reach the origin, then publish a robots.txt that confirms the same policy. Both signals matter — Cloudflare controls whether the request connects, and robots.txt tells GPTBot it is welcome to read the page once it does.
How to allow GPTBot on Cloudflare
- Open Cloudflare Dashboard → Security → WAF → Tools (or Bots).
- Add a custom rule: when User Agent contains "GPTBot", action = Skip → Bot Fight Mode and Managed Rules; then Allow.
- Repeat the same rule for PerplexityBot, ClaudeBot, Google-Extended and Bingbot.
- In robots.txt, add explicit Allow blocks for each of those user agents.
- Wait 24 hours, then verify with a fresh server log or a curl request using the GPTBot user agent.
You'll get an HTML report showing which AI crawlers are allowed or blocked on your site.
Run Express CheckRelated questions
- Cloudflare blocking AI crawlers — how to fix itBroader edge-level allowlist pattern for trusted AI bots.
- AI crawlers blocked by robots.txt — how to fix itFix policy-level disallow rules that still block bots.
- ChatGPT cannot crawl my website — how to fix itEnd-to-end checklist when ChatGPT still cannot crawl.
Updated