AI search fix
llms.txt directory: what to include and how to organise it
An llms.txt directory should be a compact, human-readable index of priority public pages, grouped into sections such as Docs, Services, and Policies. It does not replace sitemap.xml and should not list every URL on your site. The aim is to help AI systems orient quickly once crawl access works. Keep links current and remove stale routes. A well-maintained directory improves clarity but does not guarantee citations.
The usual mistake is stuffing llms.txt with hundreds of links. A smaller, curated directory is easier to maintain and often clearer for parsers.
Build an llms.txt directory in five steps
- Define three to five sections that match how visitors navigate your site.
- List only priority public URLs that stay stable over time.
- Use short labels; avoid tracking parameters in links.
- Confirm /llms.txt returns HTTP 200 as plain text or markdown.
- Review monthly and drop stale links after site changes.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on /llms.txt availability and AI crawler access on your domain.
Frequently asked questions
Should an llms.txt directory list every blog post?
Usually no. Include cornerstone pages and key resources, not the full archive.
Does Google require an llms.txt directory?
No. Google does not require llms.txt for AI features in Search.
Can a directory alone improve AI visibility?
Not by itself. Crawl access, page quality, and trust signals remain primary.
Related questions
Updated