AI search fix
Cloudflare rules for AI bots (safe)
Safe Cloudflare rules for AI bots use targeted allow or skip logic for trusted user agents — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot — while keeping broader protection for unknown traffic. Avoid blanket 'allow all bots' policies; instead align WAF behaviour with robots.txt intent, test status codes on public URLs, and confirm requests reach origin. This improves crawl readiness but does not guarantee citations in AI products.
Most regressions follow security updates: a rule added for one bot silently blocks others. Maintain one documented AI bot policy, apply it consistently across custom rules, and audit Cloudflare and origin logs after every WAF change.
Safe Cloudflare rule pattern for AI bots
- List the exact AI user agents your policy allows.
- Create WAF custom rules matching User-Agent for each trusted bot.
- Apply Skip or Allow for bot-challenge layers — not a global security bypass.
- Keep managed protections active for unknown and abusive automation.
- Validate with edge and origin logs for 200 or 304 on important HTML paths.
You'll receive an HTML report on likely edge blocks and AI crawler policy mismatches.
Run the diagnosticRelated questions
Updated