GEO Fix
Run free check

AI search fix

robots.txt for AI crawlers — how to write it

A robots.txt for AI crawlers should name each trusted user agent explicitly and state what they may fetch on your production domain. Broad `Disallow: /` rules or copied legacy templates often block GPTBot, PerplexityBot, and ClaudeBot even when you intend to welcome AI search traffic. The fix is to add separate user-agent blocks per bot, keep unknown scrapers restricted, and verify the live file at /robots.txt after deploy — especially when Cloudflare sits in front of your origin.

Start from your business policy: which AI platforms you want to read public pages for search and citation, and which broad training crawlers you want to limit. Then map each decision to a user-agent line robots parsers understand. If you use Cloudflare or another edge firewall, robots.txt must agree with those allow rules — otherwise bots see conflicting signals between the edge and your origin.

How to write robots.txt for AI crawlers

  1. List trusted bots: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot.
  2. Add one user-agent section per bot with explicit Allow paths you want indexed.
  3. Remove catch-all Disallow rules that override your AI allow intent.
  4. Deploy and fetch https://yourdomain.com/robots.txt from production, not staging.
  5. Re-test after CMS or SEO plugin updates that may rewrite robots.txt.

Free · 2 minutes · no card

See what AI crawlers hit on your site

Technical blockers, missing context, weak AI-readiness signals — in one HTML report.

No card and no payment for the audit. GEO Fix does not guarantee AI visibility or citations.

You'll get an HTML report showing which AI user agents your live robots.txt allows or blocks.

Frequently asked questions

Can one robots.txt block some AI bots and allow others?

Yes. robots.txt is evaluated per user-agent string, so you can allow GPTBot while keeping stricter rules for other crawlers.

Does robots.txt alone make my site appear in ChatGPT?

No. robots.txt only communicates crawl permission. WAF rules, content quality, and platform policies still determine whether pages are used in answers.

Related questions

Updated