AI search fix
llms.txt best practices for AI search
Good llms.txt practice begins with crawl access: robots.txt and your CDN must permit trusted AI bots before you publish /llms.txt. Keep the file short, follow llmstxt.org, link only to public 200 URLs, date your updates, and resist copying your entire sitemap. Google does not require llms.txt for AI Overviews when crawlability and helpful content are already in order. The file is optional GEO preparation for other AI systems — not a ranking lever.
Review llms.txt after pricing, product, or policy changes. Drop broken links and login-gated PDFs. Need long excerpts? Consider llms-full.txt with a named owner. Otherwise a lean map is enough. Accuracy always beats length.
llms.txt best practices checklist
- Allow GPTBot, PerplexityBot, ClaudeBot, or other bots you trust in robots.txt first.
- Use a clear H1, one-line summary, and markdown links to canonical HTML pages.
- Limit sections to what you can keep accurate within one business day of edits.
- Add an Updated line with an ISO date after each meaningful site change.
- Validate /llms.txt returns 200 over HTTPS without auth walls or redirect loops.
You'll get an HTML report on /llms.txt reachability and whether AI crawlers are blocked on your domain.
Run the diagnosticRelated questions
- What is llms.txt and do you need it for AI search?When the file helps and Google’s optional stance on AI Overviews.
- How to create an llms.txt file (with example)Step-by-step file layout at /llms.txt.
- llms.txt vs llms-full.txt: which file do you need?Short map vs longer file — choose before you over-build.
Updated