AI search fix
Cloudflare Bot Fight Mode and AI bots
Cloudflare Bot Fight Mode and Super Bot Fight Mode treat many automated clients as risky and may issue challenges or 403 responses before requests reach your origin. Trusted AI crawlers such as GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and Bingbot can be affected even when robots.txt allows them. The fix is not to remove all bot protection — create custom WAF rules that skip Bot Fight Mode and relevant managed challenges for specific AI user-agent strings, then confirm Allow in robots.txt. Turn off Block AI Bots and similar toggles if you want AI retrieval access. Allowing crawlers does not guarantee citations in ChatGPT, Copilot, or Perplexity.
GEO Fix sites often run Cloudflare proxy with Bot Fight Mode disabled and explicit allow rules instead — either approach works if logs show 200 for the bots you chose to welcome.
Allow AI bots when Bot Fight Mode is on
- Cloudflare Dashboard → Security → Bots — note whether Bot Fight Mode or Super Bot Fight Mode is enabled.
- Security → WAF → Custom rules → Create rule: User-Agent contains GPTBot (and each AI bot you allow).
- Action: Skip → Bot Fight Mode (and related bot challenge features) → then Allow or Log.
- Repeat for PerplexityBot, ClaudeBot, Google-Extended, Bingbot, OAI-SearchBot as needed.
- Publish matching Allow lines in robots.txt on origin.
- After 24 hours, filter logs for 200 responses — see how to verify AI bots in server logs.
You'll get an HTML report on robots.txt and whether common AI bots are likely blocked at the edge.
Run Express CheckRelated questions
Updated