AI search fix
ChatGPT cannot crawl my website — how to fix it
ChatGPT usually fails to crawl when GPTBot is blocked at the CDN or WAF, when robots.txt disallows it, or when both signals conflict. If either layer is closed, your pages are unavailable for OpenAI retrieval and less likely to surface in AI-generated answers. Align edge security with an explicit GPTBot and ChatGPT-User allow policy, then confirm the live domain serves stable HTML without challenge loops or redirect chains.
Work through this as a short diagnostic, not a single setting. Inspect Cloudflare or your WAF for challenges aimed at GPTBot. Read /robots.txt on the exact hostname customers use. Finally, ensure bot requests receive real HTML — not a JavaScript-only shell, infinite redirects, or repeated 403 responses from a security plugin.
Fix ChatGPT crawl access on your site
- Review Cloudflare (or your WAF) for GPTBot blocks, challenges or Bot Fight actions.
- Add edge rules that explicitly allow GPTBot and ChatGPT-User.
- Audit /robots.txt for disallow lines that conflict with your WAF policy.
- Test with access logs or controlled requests after you deploy changes.
- Monitor for 24 hours and look for repeated successful GPTBot fetches.
You'll receive an HTML report showing where ChatGPT crawlability breaks: WAF, robots.txt or both.
Run the diagnosticRelated questions
- GPTBot blocked by Cloudflare — unblock at the edgeThe most common technical barrier for ChatGPT crawl access.
- AI crawlers disallowed in robots.txtSecond gate to open once Cloudflare allows traffic through.
- llms.txt missing — optional AI site summaryUseful after crawlers can reach your pages, not instead of access fixes.
Updated