Should I allow ClaudeBot and anthropic-ai?

They may serve different flows. ClaudeBot is the primary identifier for page fetching. Read Anthropic's current documentation before permitting additional user-agent strings.

Does ClaudeBot use llms.txt?

ClaudeBot fetches HTML and standard URLs. An optional /llms.txt file may help orientation once crawl works — it cannot replace an Allow rule for the bot itself.

Will ClaudeBot traffic show in normal analytics?

Rarely in browser session tools. Use server or CDN bot dashboards instead. Clarity Bot Activity surfaces server-side AI bot requests when integrated.

AI search fix

ClaudeBot user-agent: setup and verification

ClaudeBot is the user-agent Anthropic uses when its crawler requests public HTML for Claude search and citation retrieval. Expect entries in server logs, Cloudflare AI Crawl Control and Microsoft Clarity Bot Activity when LogPush is enabled. It is distinct from Claude-Web, anthropic-ai and ordinary browser traffic. To support Claude crawl, allow ClaudeBot in robots.txt and ensure WAF rules do not return 403 or bot challenges to that string. Permitting ClaudeBot does not guarantee Claude will cite or recommend your site.

After any policy change, fetch the live robots.txt from production and compare it with your CMS editor. Combine log monitoring with a technical audit so ClaudeBot Allow rules survive the next plugin or security template update.

ClaudeBot quick reference

Operator: Anthropic — crawler for Claude retrieval workflows.
Typical log string: ClaudeBot (verify against current Anthropic publisher guidance).
Related strings: anthropic-ai, Claude-Web — set separate policy if they appear.
Policy: robots.txt Allow on public URLs plus CDN skip rules for ClaudeBot.
Verify: logs, Clarity Bot Activity, or Express Check on your domain.

You'll get an HTML report on ClaudeBot access and robots.txt policy on your domain.

Run the diagnostic

Updated 2026-05-28

ClaudeBot user-agent: setup and verification

ClaudeBot quick reference

Related questions