AI search fix
GPTBot blocked by Cloudflare — how to fix it
GPTBot is OpenAI's crawler for ChatGPT retrieval. If Cloudflare challenges or returns 403 before traffic reaches your server, GPTBot never reads your HTML — regardless of how well the site ranks in Google. Bot Fight Mode and managed WAF rules often treat GPTBot like generic scrapers unless you add an explicit skip-and-allow path for that user-agent string.
Resolve this in two layers. First, create a Cloudflare custom rule that matches User-Agent containing GPTBot and skips bot challenges before allowing the request through. Second, publish robots.txt on production with matching Allow directives so policy is consistent at the edge and origin. Re-check access logs after 24 hours; a successful fix shows 200 responses, not endless challenges.
Allow GPTBot through Cloudflare
- Sign in to Cloudflare → Security → WAF (or Bots) and review active bot protections.
- Add a custom rule: User-Agent contains GPTBot → Skip Bot Fight Mode and relevant managed rules → Allow.
- Mirror the pattern for PerplexityBot, ClaudeBot, Google-Extended and Bingbot if your policy welcomes them.
- Update live robots.txt with explicit Allow blocks for each trusted AI user agent.
- After deployment, confirm with logs or a test request using the GPTBot user-agent string.
You'll receive an HTML report showing which AI crawlers your live site allows or blocks.
Run the diagnosticRelated questions
- Targeted Cloudflare WAF rules for trusted AI botsA safer pattern than turning off all bot protection at once.
- Confirm AI crawler access in your server logsSpot 403 and challenge loops after you change WAF settings.
- GPTBot user-agent string — what to look forThe exact token to match in rules and log filters.
Updated