AI search fix
How to verify AI bots in server logs
To verify AI bots in server logs, list the user-agents you allow (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Bingbot, OAI-SearchBot), open your hosting or CDN access logs, and filter on those strings in the User-Agent field. A healthy crawl shows repeated GET requests to public URLs with HTTP 200 (or 304), not 403, 503, or endless challenge pages. Compare origin logs with Cloudflare if you use a proxy — blocks at the edge may never appear on origin. Microsoft Clarity Bot Activity helps when Cloudflare LogPush is connected, but it does not replace checking robots.txt and your WAF policy.
Log proof is crawl readiness, not citation proof. Zero GPTBot hits with thousands of Googlebot hits usually means an edge block — fix Cloudflare or robots.txt before rewriting page copy.
Verify AI bots in logs in six steps
- Write down allowed bots and one test URL (homepage or /robots.txt).
- Open access logs: hosting panel, nginx/apache, or Cloudflare → Analytics → Logs.
- Filter User-Agent contains GPTBot (repeat for each bot you care about).
- Check status codes: 200/304 on HTML paths; investigate 403/503/challenge responses.
- If edge and origin differ, fix Cloudflare WAF before tuning robots.txt on the server.
- Optional: confirm trends in Clarity Bot Activity or run Express Check on your domain.
You'll get an HTML report on robots.txt and common AI bot blocks before you dig through raw logs.
Run Express CheckRelated questions
Updated