AI search fix
Cloudflare Bot Fight Mode and AI bots
Cloudflare Bot Fight Mode and Super Bot Fight Mode treat many automated clients as risky and may issue challenges or 403 responses before traffic reaches your origin. Trusted AI crawlers such as GPTBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot can be affected even when robots.txt welcomes them. Keep broad protection on, but add custom WAF rules that skip Bot Fight for specific AI user-agent strings and match Allow lines in robots.txt. Disable Block AI Bots and similar toggles if you want retrieval access. Allowing crawlers improves fetch readiness; it does not guarantee citations in ChatGPT, Copilot or Perplexity.
Some publishers disable Bot Fight Mode entirely and rely on explicit allow lists instead — either approach works if logs show HTTP 200 for the bots you chose to welcome. The critical point is that managed robots.txt in Cloudflare edits crawl policy, while Bot Fight Mode decides whether the HTTP request succeeds at all.
Allow AI bots when Bot Fight Mode is enabled
- Cloudflare Dashboard → Security → Bots — note whether Bot Fight or Super Bot Fight Mode is on.
- Security → WAF → Custom rules → Create rule: User-Agent contains GPTBot (repeat per allowed bot).
- Action: Skip Bot Fight Mode and related challenge features, then Allow or Log.
- Repeat for PerplexityBot, ClaudeBot, Google-Extended, Bingbot and OAI-SearchBot as needed.
- Publish matching Allow lines in origin robots.txt.
- After 24 hours, filter logs for 200 responses — see how to verify AI bots in server logs.
You'll receive an HTML report on robots.txt and whether common AI bots are likely blocked at the edge.
Run the diagnosticRelated questions
Updated