AI search fix
How to verify AI bots in server logs
To verify AI bots in server logs, list the user agents you allow — GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Bingbot, OAI-SearchBot — then filter your hosting or CDN access logs on those strings in the User-Agent field. Healthy crawl activity shows repeated GET requests to public URLs with HTTP 200 or 304, not 403, 503 or endless challenge pages. Compare origin logs with Cloudflare when you use a proxy; edge blocks may never appear on origin. Microsoft Clarity Bot Activity helps when LogPush is connected, but it does not replace checking robots.txt and WAF policy.
Log proof shows crawl readiness, not citation proof. Zero GPTBot hits alongside thousands of Googlebot requests usually signals an edge block — fix Cloudflare or robots.txt before rewriting page copy.
Verify AI bots in logs in six steps
- Write down allowed bots and one test URL such as your homepage or /robots.txt.
- Open access logs: hosting panel, Nginx or Apache, or Cloudflare → Analytics → Logs.
- Filter where User-Agent contains GPTBot, then repeat for each bot you care about.
- Check status codes: 200 or 304 on HTML paths; investigate 403, 503 or challenge responses.
- When edge and origin differ, fix Cloudflare WAF before tuning robots.txt on the server.
- Optional: review trends in Clarity Bot Activity or run Express Check on your domain.
You'll receive an HTML report on robots.txt and common AI bot blocks before you dig through raw logs.
Run the diagnosticRelated questions
Updated