AI search fix
Is llms.txt actually used by ChatGPT and other AI bots?
This page is about the llms.txt file at /llms.txt on your domain — not about large language models (LLMs) as a technology. OpenAI, Anthropic, Perplexity, and other vendors do not publish a single public rule that every product always fetches llms.txt. In practice, some AI crawlers may read it after robots.txt and your CDN allow access. Google says llms.txt is not required for generative AI features in Search when crawl access and helpful content are in place.
Published studies and industry tests show mixed results: a well-formed llms.txt can orient parsers, but it does not by itself guarantee citations in ChatGPT or Perplexity. Treat the file as optional GEO preparation after GPTBot, ClaudeBot, and PerplexityBot can fetch your HTML. If bots are blocked at the edge, llms.txt usage is irrelevant until you fix crawl policy.
Practical order for llms.txt and AI bots
- Allow trusted AI crawlers in robots.txt and remove WAF false positives.
- Confirm priority URLs return 200 HTML without login walls.
- Publish a short, accurate /llms.txt with links you want emphasized.
- Monitor server logs for GPTBot or PerplexityBot fetches — including /llms.txt if listed.
You'll get an HTML report showing whether /llms.txt exists and whether robots.txt or WAF rules block AI crawlers.
Run Express CheckRelated questions
Updated