AI search fix
Server rules for AI bots: Cloudflare, Nginx and how to verify access in logs
AI crawlers most often get blocked at the server or CDN layer rather than in robots.txt: Cloudflare Bot Fight Mode and Managed Rules bundle GPTBot, PerplexityBot, ClaudeBot and similar bots into generic automation heuristics by default, and Nginx or Apache configs frequently inherit deny rules from old templates that catch trusted bots along with real scrapers. The fix is the same shape on every stack — add explicit, narrow allow or skip rules for the exact user-agent strings you trust, keep default protection for unknown automation, then confirm the change worked by filtering server or CDN logs for HTTP 200 or 304 responses rather than assuming a saved rule is a working one. None of this guarantees AI citations; it only confirms bots can reach your pages.
Most regressions follow a security change, not a content change: a rule fixes one bot and silently blocks another, or an inherited config include reintroduces a deny rule a later edit never touched. Keep one documented AI bot policy and re-verify logs after every WAF, plugin or Nginx config deploy.
Server-layer blocks at a glance
| Layer | What usually goes wrong |
|---|---|
| Cloudflare Bot Fight Mode / Managed Rules | Treats AI crawlers as generic automation and issues a 403 or JS challenge with no explicit rule ever written against them. |
| Cloudflare account-level "Block AI Bots" | Overrides custom WAF allow rules until it is turned off — a skip rule alone will not fix the block. |
| Nginx or Apache config | Inherited deny rules in included or old server blocks catch trusted bots even after the visible config looks correct. |
| Server logs read alone | Origin-only logs miss edge blocks entirely — a request Cloudflare rejects never reaches the server's own log. |
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on likely Cloudflare, Nginx and robots.txt blocks for AI crawlers on your domain.
Cloudflare: the most common block, and the two-gate model
Cloudflare's Bot Fight Mode and Managed Rules bundle AI crawlers into generic automation heuristics by default, so GPTBot, PerplexityBot, ClaudeBot, Google-Extended or Bingbot can receive a 403 or a JS challenge with no explicit rule ever written against them. Think of Cloudflare as the first gate and robots.txt as the second: if Cloudflare blocks first, the crawler never reaches robots.txt at all, and if Cloudflare allows but robots.txt disallows, the crawler still will not process the page for search and citation.
In Cloudflare Dashboard → Security → WAF → Custom rules, add a rule that matches User-Agent contains GPTBot, action Skip for Bot Fight Mode and Managed Rules, then Allow — and repeat the same pattern for each bot you trust rather than assuming one rule covers all of them. Check Security → Bots for an account-level "Block AI Bots" or Crawler Protection toggle too: it can override a correct custom WAF rule until it is switched off.
A safe Cloudflare rule pattern: allow specific bots, not all bots
Avoid a blanket "allow all bots" policy. A safer pattern lists the exact AI user-agents your policy welcomes — for example GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot — and applies Skip or Allow only for the bot-challenge layers those strings hit, while keeping managed protection active for unknown or abusive automation. Managed robots.txt in Cloudflare edits crawl policy; Bot Fight Mode controls whether the HTTP request succeeds at all — the two are different layers, and both need to agree.
Publish the matching Allow lines in robots.txt on origin so the edge and the crawl-policy layer agree, then wait roughly 24 hours before judging the result — propagation and re-crawl are not instant.
Nginx: audit inherited config before adding one allow rule
Many Nginx setups inherit copy-pasted deny rules from old hardening templates that catch trusted bots along with real scrapers — review every included and inherited server block before changing a single location directive, since a hidden include can still return 403 to GPTBot after the visible config looks fixed. Map each trusted user-agent explicitly, in a map block or a documented set of conditions, rather than one brittle catch-all pattern that is hard to audit later.
Nginx should return normal 200 responses to allowed bots on public HTML routes while keeping rate limits and abuse controls for unknown traffic. After any config reload, test with curl -A GPTBot against a real page and confirm the status code in access logs — do not trust a config that "looks right" without a live request behind it.
Verify the fix in logs — six steps, not one assumption
Log proof is crawl readiness, not citation proof, but it is the only real confirmation a fix worked. Write down the bots you allow and one test URL, open access logs — hosting panel, Nginx or Apache, or Cloudflare → Analytics → Logs — and filter where User-Agent contains the exact string, repeating per bot rather than a loose partial match that blends several crawlers into one bucket.
Check status codes: 200 or 304 on HTML paths is healthy; 403, 503 or an endless challenge page means a WAF or CDN block, not a robots.txt problem. When edge and origin logs disagree, fix the Cloudflare rule before touching robots.txt again — zero GPTBot hits alongside thousands of Googlebot hits after a deploy or WAF change is a red flag worth comparing against the previous week.
Frequently asked questions
Should I disable Cloudflare Bot Fight Mode entirely?
Not necessarily. A safer default for most sites is to keep Bot Fight Mode on and add targeted skip rules for the trusted AI user-agents you explicitly allow.
Does Cloudflare "Block AI Bots" override my custom WAF rules?
Yes. If the account-level Block AI Bots or Crawler Protection toggle is on, it can block AI user-agents even when a correct custom WAF skip rule exists — turn the toggle off for retrieval access.
Is Managed robots.txt in Cloudflare the same thing as Bot Fight Mode?
No. Managed robots.txt edits crawl policy; Bot Fight Mode controls whether the HTTP request succeeds at all. They are different layers, and both need to agree with your intent.
Can robots.txt alone fix a Cloudflare block?
No. robots.txt is advisory once access is granted. If Cloudflare blocks a request at the edge, the crawler never reaches robots.txt to read it.
Should I rely only on user-agent matching in Nginx?
It is a practical start, but pair it with ongoing log monitoring and broader abuse controls rather than treating a user-agent allowlist as the entire policy.
How do I test a bot rule quickly?
Run curl with a bot user-agent, for example curl -A GPTBot against a real page, and confirm the status code. It catches obvious WAF or Nginx blocks fast, though it does not replace real crawler visits.
How often should AI bots appear in server logs?
There is no fixed quota. Sudden zero traffic for a bot that used to appear regularly, right after a deploy or WAF change, is the signal worth investigating.
Does a working server rule guarantee AI citations?
No. A correct Cloudflare or Nginx rule only proves a bot can fetch your pages. Whether a platform cites or recommends them still depends on content quality and each platform's own rules.
Related questions
- AI crawlers blocked — how to fix GPTBot, PerplexityBot, ClaudeBot and Google-ExtendedDiagnose robots.txt and CDN/WAF blocks per bot, plus the Google-Extended policy decision.
- robots.txt for AI crawlers — how to write itTemplate for trusted AI user-agent sections.
- Bingbot and Copilot: Are You Blocking Microsoft's AI?How Microsoft crawl and grounding flows connect to site visibility.
Updated