AI search fix

What is llms.txt and do you need it for AI search?

llms.txt is a plain-text or markdown file at https://yourdomain.com/llms.txt that summarizes what your site offers and links to priority pages. The llmstxt.org proposal targets LLM workflows, not traditional search rankings. Google says you do not need llms.txt for generative AI features in Search when crawl access and helpful content are in place. For ChatGPT, Perplexity, Claude and similar tools, publishing a lightweight llms.txt can still be a practical GEO step after robots.txt and WAF rules allow AI crawlers.

Treat llms.txt as an optional briefing layer, not a second sitemap: a title, summary and links to the pages you most want AI systems to notice. It does not replace robots.txt, schema.org, or strong on-page answers, and it does not need to list every URL on your site.

llms.txt best practices checklist

Allow GPTBot, PerplexityBot, ClaudeBot and other AI bots you trust in robots.txt before you publish the file.
Use the llmstxt.org layout: H1 site name, one-line summary, then markdown links grouped under headings like Docs, Services, Policies.
Link only priority public 200 URLs you can keep accurate — not a dump of your full sitemap.
Add an Updated line with an ISO date and refresh it after pricing, product or route changes.
Validate that /llms.txt returns 200 over HTTPS as plain text or markdown, with no auth wall or redirect loop.

Free · 2 minutes · no card

See what AI crawlers hit on your site

Technical blockers, missing context, weak AI-readiness signals — in one HTML report.

You'll get an HTML report showing whether /llms.txt exists and whether robots.txt or WAF rules block AI crawlers.

Why llms.txt was proposed

The llms.txt idea comes from the llmstxt.org proposal, which argues that large language models work with limited context windows and benefit from a short, curated guide to a site rather than crawling and parsing every page. A conventional website mixes navigation, scripts and marketing markup that is noisy for a model to read. An llms.txt offers a clean index instead: what the site is, and where the most important content lives. It is a community proposal, not a standard backed by search engines.

Does it actually work today, and does it help SEO?

The picture is mixed. Google has said llms.txt is not required for its generative AI features in Search and has not confirmed using the file — it is not a ranking factor like backlinks or Core Web Vitals. The major AI assistants have not publicly documented reading it either. What does use it today is a growing set of LLM tools, agents and documentation pipelines that fetch llms.txt to orient before crawling individual pages.

So the file can help specific workflows, but claims that it lifts rankings or guarantees citations are not supported. Pair any llms.txt work with fixing crawl access and adding real schema and content — llms.txt cannot compensate for blocked bots or thin pages.

llms.txt vs llms-full.txt vs robots.txt vs sitemap.xml

llmstxt.org also describes llms-full.txt, an optional longer sibling that can hold expanded excerpts for the same priority pages. Most small sites need only the short llms.txt; add llms-full.txt only when you have stable long-form docs and a real owner to keep excerpts from drifting out of sync with the live pages they summarize.

robots.txt, sitemap.xml and llms.txt answer different questions. robots.txt sets access policy — which crawlers may fetch which paths. sitemap.xml helps search engines discover every indexable URL. llms.txt is neither a gate nor a full inventory; it is a short editorial map. robots.txt must allow AI crawlers first, sitemap.xml aids discovery, and llms.txt provides orientation once a system is already reading your site.

What to put in it, and what to leave out

Build the file as a compact directory, not a link dump: an H1 site name, a one-line summary, then three to five sections — commonly Docs, Services and Policies — each listing only the public URLs you can keep current. Skip login-gated pages and tracking parameters; a smaller, curated file is easier to maintain than hundreds of links copied from your navigation.

Should you add one?

For most sites it is low-cost and low-risk, so publishing a lean llms.txt is reasonable once crawl access is solid and someone can keep it current. It will not compensate for blocked crawlers, thin pages or missing schema, and it will not, by itself, get your brand named in ChatGPT or Perplexity. Add it as one small part of GEO hygiene — not a shortcut — and measure results against real visibility, not the mere presence of the file.

Frequently asked questions

Is llms.txt required for ChatGPT or Perplexity?

No. Crawl access through robots.txt and your CDN or host comes first. llms.txt may help some pipelines orient faster once pages are reachable.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short map of priority links; llms-full.txt is an optional, longer sibling with expanded excerpts. Most sites need only the short file.

How long should llms.txt be?

Usually one screen of markdown covering key services, docs and policies. If it grows past routine maintenance, trim it or split detail into HTML pages.