AI search fix
Is llms.txt actually used by ChatGPT and other AI bots?
This page covers the llms.txt file at /llms.txt — not large language models (LLMs) as a technology. OpenAI, Anthropic, Perplexity and others publish no single rule that every product always fetches llms.txt. Some AI crawlers may request it once robots.txt and your CDN permit access, but behaviour varies by platform. Google’s position on AI Overviews is separate: llms.txt is optional there; crawl access and helpful content come first.
Industry tests show mixed results. A well-formed llms.txt can help parsers orient, yet it alone will not guarantee citations in ChatGPT or Perplexity. Treat the file as optional GEO preparation only after GPTBot, ClaudeBot, and PerplexityBot can fetch your HTML. If bots are blocked at the edge, llms.txt is irrelevant until crawl policy is fixed.
Practical order for llms.txt and AI bots
- Allow trusted AI crawlers in robots.txt and remove WAF false positives.
- Confirm priority URLs return 200 HTML without login walls.
- Publish a short, accurate /llms.txt with links you want emphasised.
- Monitor server logs for GPTBot or PerplexityBot fetches — including /llms.txt if listed.
You'll get an HTML report showing whether /llms.txt exists and whether robots.txt or WAF rules block AI crawlers.
Run the diagnosticRelated questions
Updated