AI search fix
ClaudeBot blocked — how to fix it
ClaudeBot is Anthropic's crawler for fetching pages used in Claude search and citation workflows. When robots.txt disallows it or your edge firewall challenges automation traffic, Claude cannot read your site reliably. Publish an explicit ClaudeBot allow policy in production robots.txt, match it with Cloudflare or WAF skip rules for the exact user-agent string, and verify live responses after deployment.
Many sites block ClaudeBot by accident through all-bot disallow snippets or aggressive bot-management defaults. Give ClaudeBot its own user-agent section, restrict disallow to paths you genuinely want private, and skip Bot Fight for that string at the edge. After changes, confirm the hostname your customers use serves the same robots.txt your CMS dashboard displays.
Unblock ClaudeBot on your site
- Review /robots.txt for ClaudeBot, anthropic-ai or global disallow entries.
- Add a ClaudeBot user-agent block with Allow paths for public marketing and docs pages.
- Create a Cloudflare custom rule to allow or skip challenges for ClaudeBot.
- Deploy and test production robots.txt plus response headers on sample URLs.
- Check logs after 24 hours for successful ClaudeBot requests.
You'll receive an HTML report on whether ClaudeBot is blocked in robots.txt or at your WAF.
Run the diagnosticRelated questions
Updated