AI search fix
llms.txt missing — how to fix it
When llms.txt is missing at your domain root, AI workflows lack a standard short briefing about your brand, services, and priority URLs. Crawlers may still fetch pages if robots.txt and your WAF allow them, but assistants infer structure from HTML alone. Google says llms.txt is not required for generative AI in Search when crawlability and helpful content are solid. For ChatGPT, Perplexity, Claude and similar systems, publishing /llms.txt is still practical GEO: markdown with a one-line summary, brief context, and linked sections for important pages. Once you add the file, a free llms.txt validator confirms the structure and links parse before crawlers reach it.
llms.txt does not replace robots.txt, Cloudflare allow rules, or strong on-page content. It complements technical access with a readable map some LLM pipelines use. Keep it lightweight per llmstxt.org. Regenerate when pricing, routes, or core messaging change.
How to add llms.txt when it is missing
- Open https://yourdomain.com/llms.txt and confirm 404 or an empty response.
- Draft H1 title, blockquote summary, and a short context paragraph.
- Add sections such as Docs, Services, and Policies with markdown list links.
- Serve at /llms.txt with Content-Type text/plain or text/markdown.
- Add an Updated date line and re-check after your next deploy.
Free · 2 minutes · no card
Does your site have a working llms.txt?
The free Express Check reads your llms.txt, robots.txt and schema and shows what's missing.
You'll get an HTML report on /llms.txt reachability and whether AI crawlers are blocked.
What an llms.txt file actually does
An llms.txt file is a small markdown document at your domain root that gives AI tools a fast, human-readable map of your site: who you are, what you offer, and which URLs matter most. It is a proposed convention, not an official standard, so support varies between systems. Google has stated that llms.txt is not required to appear in its generative AI features, and several engineers note that major assistants do not yet read the file directly. Treat it as one orientation signal among many, not a switch that turns AI visibility on.
Where it can help is the growing set of LLM-based tools and pipelines that do fetch a site llms.txt to summarise scope before crawling individual pages. Publishing one is low-cost and low-risk: at worst it is ignored, at best it gives compatible systems a cleaner brief than parsing your full HTML. Keep expectations grounded, though — a tidy llms.txt cannot compensate for blocked crawlers or thin pages.
Why your llms.txt is missing or returns 404
If https://yourdomain.com/llms.txt returns a 404, the usual reasons are simple. The file was never created; it was uploaded to a subfolder instead of the web root; or the platform routes unknown paths to a catch-all page rather than serving a static file. On some stacks the file exists but is sent with the wrong Content-Type, so browsers download it instead of showing plain text, and some checks treat that as missing. A stale CDN cache, an over-broad redirect, or a WAF challenge can also hide a file that is technically present.
Diagnose it in order: request the URL directly and read the raw status code, confirm the file sits at the root and not under a build folder in production, then inspect the response headers. Fixing the delivery is often faster than rewriting the file itself.
How to write an llms.txt worth fetching
Follow the format described at llmstxt.org. Start with an H1 that names the site, add a one-line blockquote summary, then a short paragraph of context. Below that, organise links under clear markdown headings such as Docs, Services, and Optional, pointing to the pages you most want represented. Keep it lightweight — plain markdown headings and lists, no HTML tables, scripts, or tracking. Many teams generate a first draft with a plugin or generator, then trim it by hand so it reads like a briefing rather than a sitemap dump.
If your documentation is extensive, you can also publish an llms-full.txt with expanded content while keeping llms.txt itself short. Whichever you choose, write for a reader skimming in seconds: concrete names, plain language, and links that resolve.
After publishing: validate and stay realistic
Once the file is live, confirm it returns a 200 response with a text/plain or text/markdown Content-Type, then re-check after your next deploy so a build step does not quietly drop it. A quick manual fetch or a validator is enough; you do not need a dedicated tool. Add an Updated line and refresh the file whenever pricing, routes, or core messaging change, so AI-facing summaries do not drift from reality.
Finally, keep llms.txt in perspective. Crawl access through robots.txt and your WAF, plus substantive pages that answer real questions, do more for AI visibility than any single file. An llms.txt can make compatible systems orient faster, but it does not guarantee that ChatGPT, Perplexity, or Google will cite you — inclusion is decided by each platform.
Frequently asked questions
Does llms.txt replace robots.txt?
No. robots.txt controls crawler access. llms.txt explains what the site offers once access is allowed.
Do I need llms.txt for Google AI Overviews?
No. Google documentation states llms.txt is not required for generative AI features in Search when crawl and content are solid.
Will adding llms.txt guarantee AI citations?
No. llms.txt may help orientation; citations still depend on quality, relevance, and platform policy.
Related questions
Updated