AI search fix
WordPress blocking AI crawlers — how to fix it
WordPress sites frequently block AI crawlers through security plugins, cached robots.txt overrides, or Cloudflare rules copied from generic hardening guides. GPTBot, PerplexityBot, and ClaudeBot may receive 403 responses even when Googlebot works. The fix is to audit plugins such as Wordfence, review the virtual or physical robots.txt output, and add explicit allow rules for the AI user agents you trust.
Check both the Settings → Reading “discourage search engines” flag and SEO plugin robots editors — they stack. After changes, fetch robots.txt and a sample post URL with a GPTBot user agent from CLI or logs.
How to fix WordPress AI bot blocks
- Confirm “discourage search engines” is off in WordPress settings.
- Review security plugin bot blocking lists and allow trusted AI agents.
- Inspect Yoast, Rank Math, or other SEO plugin robots.txt overrides.
- Align Cloudflare or host WAF rules with your robots.txt policy.
- Validate JSON-LD output from your SEO plugin on key pages.
You'll get an HTML report on WordPress robots.txt, plugin blocks, and schema coverage.
Run Express CheckRelated questions
- Cloudflare blocking AI crawlers — how to fix itBroader edge-level allowlist pattern for trusted AI bots.
- AI crawlers blocked by robots.txt — how to fix itFix policy-level disallow rules that still block bots.
- Schema markup missing for AI search — how to fix itAdd JSON-LD so pages are easier to parse and cite.
Updated