AI search fix
How to set up llms.txt for Claude (ClaudeBot crawl first)
Setting up llms.txt for Claude begins with crawl access: ClaudeBot must be allowed in robots.txt and not blocked by Cloudflare or Wordfence. Then publish a lean /llms.txt using the llmstxt.org layout with links to priority public pages. Anthropic documents ClaudeBot as a crawler; llms.txt is optional orientation — not a switch that forces Claude to cite you. Verify with server logs or Bot Activity after fixes.
If Claude cannot fetch your homepage, fix claudebot-blocked issues before editing llms.txt. Keep the file aligned with what human visitors see — no keyword stuffing or hidden URLs.
Claude llms.txt setup in four steps
- Check robots.txt: ClaudeBot is not disallowed on URLs you need discovered.
- Review CDN or WAF: no Bot Fight Mode or custom rules blocking ClaudeBot.
- Publish /llms.txt with H1, summary, and markdown links to docs, services, and policies.
- Request https://yourdomain.com/llms.txt in a browser — expect 200 over HTTPS without auth.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on ClaudeBot access and /llms.txt reachability on your domain.
Frequently asked questions
Does Claude always read llms.txt?
No platform guarantees that. ClaudeBot crawls allowed URLs; llms.txt may help orientation when present and accurate. Crawl access and clear HTML come first.
Is llms.txt required for Claude?
No. It is optional GEO preparation. Required work is allowing ClaudeBot and serving clear public pages with sound schema where relevant.
How do I know ClaudeBot reached my site?
Check origin or CDN logs for ClaudeBot, or Microsoft Clarity Bot Activity when Cloudflare LogPush is connected. Express Check flags common robots.txt blocks.
Related questions
Updated