AI search fix
How to set up llms.txt for Claude (ClaudeBot crawl first)
Setting up llms.txt for Claude starts with crawl access: ClaudeBot must be allowed in robots.txt and not blocked by Cloudflare or Wordfence rules. Then publish a lean /llms.txt using the llmstxt.org layout with links to your priority public pages. Anthropic documents ClaudeBot as a crawler; llms.txt is optional orientation for AI systems — not a switch that forces Claude to cite you. Verify with server logs or Bot Activity after fixes.
If Claude cannot fetch your homepage, fix claudebot-blocked issues before editing llms.txt. Keep the file aligned with what you show human visitors — no keyword stuffing or hidden URLs.
Claude llms.txt setup in four steps
- Check robots.txt: ClaudeBot user-agent is not disallowed on pages you need indexed.
- Review CDN/WAF: no Bot Fight Mode or custom rules blocking ClaudeBot.
- Publish /llms.txt with H1, summary, and markdown links to docs, services, and policies.
- Request https://yourdomain.com/llms.txt in a browser — expect 200 over HTTPS without auth.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on ClaudeBot access and /llms.txt reachability on your domain.
Frequently asked questions
Does Claude always read llms.txt?
No platform guarantees that. ClaudeBot crawls allowed URLs; llms.txt may help orientation when present and accurate. Crawl access and helpful HTML content come first.
Is llms.txt required for Claude?
No. It is optional GEO preparation. Required work is allowing ClaudeBot and serving clear public pages with solid schema where relevant.
How do I know ClaudeBot reached my site?
Check server or CDN logs for ClaudeBot user-agent, or Microsoft Clarity Bot Activity if Cloudflare LogPush is connected. Express Check flags common robots.txt blocks.
Related questions
Updated