AI search fix
llms.txt best practices for AI search
llms.txt best practices start with crawl access: robots.txt and your CDN must allow trusted AI bots before you publish /llms.txt. Keep the file short, use the llmstxt.org layout, link only to public 200 URLs, date updates, and avoid duplicating your entire sitemap. Google does not require llms.txt for generative AI in Search when crawlability and helpful content are solid. The file is optional GEO preparation for other AI systems — not a ranking switch.
Review llms.txt after pricing, product, or policy changes. Remove broken links and login-gated PDFs. If you need long excerpts, consider llms-full.txt with an owner — otherwise stay with a lean map.
llms.txt best practices checklist
- Allow GPTBot, PerplexityBot, ClaudeBot, or other trusted bots in robots.txt first.
- Use a clear H1, one-line summary, and markdown links to canonical HTML pages.
- Limit sections to what you can keep accurate within one business day of edits.
- Add an Updated line with an ISO date after each meaningful site change.
- Validate /llms.txt returns 200 over HTTPS without auth walls or redirect loops.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on /llms.txt reachability and AI crawler access.
Frequently asked questions
How long should llms.txt be?
Usually one screen of markdown — key services, docs, and policies. Split detail into HTML if maintenance becomes heavy.
Should llms.txt repeat my full navigation menu?
No. Link only URLs you want AI systems to prioritise. Accuracy beats length.
Do llms.txt best practices guarantee AI citations?
No. They improve orientation after crawlers can read your site. Citations still depend on relevance, trust, and platform policy.
Related questions
Updated