AI search fix
AI crawlers blocked by robots.txt — how to fix it
When robots.txt disallows GPTBot, PerplexityBot or ClaudeBot, your site becomes unreadable for AI search workflows even though pages stay public in Google. Broad `Disallow: /` lines and copied legacy templates are the usual cause. Fix it by writing explicit per-bot sections on production, removing catch-all rules that override your intent, and fetching https://yourdomain.com/robots.txt after every CMS or SEO plugin update.
Decide which platforms may read your marketing and product pages: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Bingbot are the common set. Give each a dedicated user-agent block with clear Allow paths. Keep stricter rules for unknown scrapers in separate sections so you do not weaken overall abuse protection.
Repair robots.txt for AI crawlers
- Copy the live file from /robots.txt on your production hostname — not staging.
- Remove or narrow disallow rules that block trusted AI user agents.
- Add explicit sections for GPTBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot.
- Deploy and re-fetch the file from production to confirm what visitors receive.
- Check server logs to verify trusted bots get allowed responses, not 403 from a WAF.
You'll receive an HTML report listing which AI user agents your live robots.txt disallows.
Run the diagnosticRelated questions
Updated