Should I allow ClaudeBot and anthropic-ai?

They may serve different flows. ClaudeBot is the primary crawl identifier for page fetching. Review Anthropic documentation before allowing additional user-agent strings.

Does ClaudeBot use llms.txt?

ClaudeBot fetches HTML and standard URLs. Optional /llms.txt may help orientation after crawl works — it is not a substitute for allowing the bot.

Will ClaudeBot traffic show in normal analytics?

Usually not in browser session tools. Use server or CDN bot dashboards. Clarity Bot Activity shows server-side AI bot requests when integrated.

AI search fix

ClaudeBot user-agent: setup and verification

ClaudeBot is the user-agent Anthropic uses when its crawler requests public HTML for Claude search and citation-oriented retrieval. You will see it in server logs, Cloudflare AI Crawl Control, and Microsoft Clarity Bot Activity when LogPush is connected. It is not the same as Claude-Web, anthropic-ai, or end-user browser traffic. To support Claude crawl, allow ClaudeBot in robots.txt and avoid WAF rules that return 403 or challenges to that string. Allowing ClaudeBot does not guarantee Claude will cite or recommend your site.

After policy changes, verify the live robots.txt on your production host matches what your CMS shows. Pair log checks with a technical audit so ClaudeBot allow rules are not undone by the next plugin update.

ClaudeBot quick reference

Operator: Anthropic — crawler for Claude retrieval workflows.
Typical log string: ClaudeBot (confirm in current Anthropic publisher guidance).
Related strings: anthropic-ai, Claude-Web — policy separately if they appear.
Policy: robots.txt Allow for public URLs + CDN skip rules for ClaudeBot.
Verify: logs, Clarity Bot Activity, or Express Check on your domain.

You'll get an HTML report on ClaudeBot access and robots.txt policy on your domain.

Run Express Check

Updated 2026-05-20

ClaudeBot user-agent: setup and verification

ClaudeBot quick reference

Related questions