AI search fix
llms.txt directory: what to include and how to organize it
An llms.txt directory should be a compact, human-readable index of priority public pages, grouped by sections like Docs, Services, and Policies. It is not a replacement for sitemap.xml and should not include every URL on your site. The goal is to help AI systems orient quickly after crawl access works. Keep links current and remove stale routes. A good directory improves clarity, but it does not guarantee citations.
The most common mistake is overloading llms.txt with hundreds of links. A smaller, curated directory is usually more useful for both maintenance and machine parsing.
Build an llms.txt directory in five steps
- Define 3-5 sections that reflect how users navigate your site.
- List only priority public URLs that stay stable over time.
- Use short labels and avoid tracking parameters in links.
- Validate /llms.txt returns 200 and plain text or markdown.
- Review monthly and remove stale links after site updates.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on /llms.txt availability and crawler access on your domain.
Frequently asked questions
Should llms.txt directory include all blog posts?
Usually no. Include cornerstone pages and key resources, not every URL.
Is llms.txt directory required by Google?
No. Google does not require llms.txt for AI features in Search.
Can a directory improve AI visibility by itself?
Not by itself. Crawl access, page quality, and trust signals remain primary.
Related questions
Updated
