GEO FixAI search readiness check

AI search fix

robots.txt for AI crawlers — how to write it

robots.txt for AI crawlers should name each trusted user-agent explicitly and state what they may fetch on your production domain. Broad Disallow: / rules or legacy templates often block GPTBot, PerplexityBot, and ClaudeBot even when you want AI search traffic. Add separate user-agent blocks per bot, keep unknown scrapers restricted, and verify the live file at /robots.txt after deploy — especially when Cloudflare sits in front of your origin.

Start from policy: which AI platforms may read public pages for search and citation, and which training crawlers you limit. Map each decision to user-agent lines parsers understand. If you use Cloudflare or another edge firewall, robots.txt must agree with allow rules — otherwise bots see conflicting edge and origin signals.

How to write robots.txt for AI crawlers

  1. List trusted bots: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot.
  2. Add one user-agent section per bot with explicit Allow paths you want indexed.
  3. Remove catch-all Disallow rules that override your AI allow intent.
  4. Fetch https://yourdomain.com/robots.txt from production, not staging.
  5. Re-test after CMS or SEO plugin updates that may rewrite robots.txt.

Free · 2 minutes · no card

See what AI crawlers hit on your site

Technical blockers, missing context, weak AI-readiness signals — in one HTML report.

No card and no payment for the free check. Prices for Starter and Pro exclude VAT as shown on the site. GEO Fix does not guarantee AI visibility or citations.

You'll get an HTML report showing which AI user-agents your live robots.txt allows or blocks.

Frequently asked questions

Can one robots.txt block some AI bots and allow others?

Yes. robots.txt is evaluated per user-agent string, so you can allow GPTBot whilst keeping stricter rules for other crawlers.

Does robots.txt alone make my site appear in ChatGPT?

No. robots.txt communicates crawl permission. WAF rules, content quality, and platform policies still decide answer use.

Should robots.txt mention every AI bot?

Document each bot your policy cares about. Undocumented bots may still be blocked at the edge even when robots.txt is silent.

Related questions

Updated