robots.txt for AI crawlers — how to write it

A robots.txt for AI crawlers should name each trusted user agent explicitly and state what they may fetch on your production domain. Broad `Disallow: /` rules or copied legacy templates often block GPTBot, PerplexityBot, and ClaudeBot even when you intend to welcome AI search traffic. The fix is to add separate user-agent blocks per bot, keep unknown scrapers restricted, and verify the live file at /robots.txt after deploy — especially when Cloudflare sits in front of your origin.

Start from your business policy: which AI platforms you want to read public pages for search and citation, and which broad training crawlers you want to limit. Then map each decision to a user-agent line robots parsers understand. If you use Cloudflare or another edge firewall, robots.txt must agree with those allow rules — otherwise bots see conflicting signals between the edge and your origin.