AI search fix
Cloudflare blocking AI crawlers — how to fix it
Cloudflare may stop GPTBot, PerplexityBot, ClaudeBot and similar crawlers when Bot Fight Mode or managed rules classify them as risky automation. If the edge returns 403 or a JavaScript challenge, AI assistants cannot fetch your public pages — so your content is less likely to appear in AI-generated answers. Create explicit allow or skip rules for the user agents you trust, then confirm robots.txt on the live domain does not contradict that policy.
Picture two gates: Cloudflare at the edge, robots.txt at the origin. Both must agree for the bots you want. If Cloudflare blocks first, the crawler never reaches robots.txt; if Cloudflare allows but robots.txt disallows, the bot still will not process your pages for search workflows. Document which AI platforms you welcome, apply the same list in WAF and robots.txt, and validate with logs after each security change.
Stop Cloudflare blocking trusted AI crawlers
- Open Cloudflare Dashboard → Security → WAF → Custom rules.
- Create separate rules for GPTBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot user agents.
- Set the action to allow or skip bot-challenge features for those strings only.
- Keep anti-abuse protections for unknown bots and aggressive scrapers.
- Review access logs after 24 hours to confirm successful fetches on key URLs.
You'll receive an HTML report on whether your WAF or robots policy blocks trusted AI crawlers.
Run the diagnosticRelated questions
Updated