AI crawlers blocked by robots.txt — how to fix it

When robots.txt disallows trusted AI bots, your pages become unreadable for AI search workflows even if they are public in Google. This usually happens due to broad `Disallow: /` rules or legacy bot-blocking snippets copied from old SEO templates. To fix it, define clear allow policy per user agent and test the final file in production, not only in staging.

Start by listing the bots you actually want to serve: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, and Bingbot. Then write explicit user-agent sections for those bots and avoid catch-all rules that accidentally override your intent. Keep your anti-scraper policy for unknown bots separately so you do not lose security coverage.