AI search fix
GPTBot blocked by Cloudflare — how to fix it
GPTBot is OpenAI's web crawler for ChatGPT. If Cloudflare's WAF blocks it, your pages stay invisible to ChatGPT even when they rank well on Google. By default Cloudflare's bot fight rules treat GPTBot as a threat and silently return a 403 or a JS challenge, so the answer never includes you.
The fix is two-step: explicitly allow GPTBot at the Cloudflare edge so requests reach the origin, then publish a robots.txt that confirms the same policy. Both signals matter — Cloudflare controls whether the request connects, and robots.txt tells GPTBot it is welcome to read the page once it does.
How to allow GPTBot on Cloudflare
- Open Cloudflare Dashboard → Security → WAF → Tools (or Bots).
- Add a custom rule: when User Agent contains "GPTBot", action = Skip → Bot Fight Mode and Managed Rules; then Allow.
- Repeat the same rule for PerplexityBot, ClaudeBot, Google-Extended and Bingbot.
- In robots.txt, add explicit Allow blocks for each of those user agents.
- Wait 24 hours, then verify with a fresh server log or a curl request using the GPTBot user agent.
You'll get an HTML report showing which AI crawlers are allowed or blocked on your site.
Run the free checkRelated questions
- Cloudflare rules for AI bots (safe)Targeted WAF allow and skip rules for trusted AI user-agents.
- How to verify AI bots in server logsLog-based validation steps for trusted AI bot access.
- GPTBot user-agent: what it is and how to allow itHow to identify GPTBot and separate it from other OpenAI user-agents.
Updated