AI search fix
ClaudeBot blocked — how to fix it
ClaudeBot is Anthropic's web crawler used to fetch pages for Claude search and citation workflows. When robots.txt disallows it or your edge firewall challenges automation traffic, Claude cannot read your site reliably. The fix is to publish an explicit ClaudeBot allow policy in robots.txt, match that policy in Cloudflare or your WAF, and verify live responses after deployment.
Many sites block ClaudeBot accidentally through all-bot disallow snippets or aggressive bot management defaults. Treat ClaudeBot like other trusted AI crawlers: dedicated user-agent section, narrow disallow only where needed, and edge rules that skip bot fight for the exact user-agent string. After changes, confirm the production domain serves the same robots.txt your CMS dashboard shows.
How to fix ClaudeBot blocked
- Review /robots.txt for ClaudeBot, anthropic-ai, or global disallow entries.
- Add a ClaudeBot user-agent block with Allow paths for public marketing and docs pages.
- Create a Cloudflare custom rule to allow or skip challenges for ClaudeBot.
- Deploy and test the production robots.txt and response headers.
- Check logs after 24 hours for successful ClaudeBot requests.
You'll get an HTML report showing whether ClaudeBot is blocked in robots.txt or at your WAF.
Run Express CheckRelated questions
Updated