Cloudflare blocking AI crawlers — how to fix it

Cloudflare can block trusted AI crawlers when Bot Fight Mode or Managed Rules treat them like generic automation traffic. If that happens, AI assistants cannot fetch your pages, so your site is less likely to be cited in AI answers. The fix is to create explicit allow rules for trusted user agents and verify that robots.txt does not contradict your WAF policy.

Think of Cloudflare as the first gate and robots.txt as the second gate. You need both open for the bots you want: if Cloudflare blocks first, the crawler never reaches robots.txt; if Cloudflare allows but robots.txt disallows, the crawler still will not process the page content for search and citation workflows.