AI search fix
Google-Extended blocked — should you allow it?
If Google-Extended is blocked in your robots.txt, Google's generative AI crawler may not fetch your pages for Gemini-related use cases, whilst Googlebot can still crawl for classic Search indexing. Whether you should allow Google-Extended depends on legal policy, content licensing, and appetite for Google's generative AI systems to read public pages — not on a guarantee of more AI Overviews or citations. Allowing Google-Extended does not force inclusion; blocking it does not automatically protect all training uses if other crawlers or channels apply.
Document the decision for your team: allow both Googlebot and Google-Extended, block only Google-Extended, or seek counsel for regulated industries. Re-fetch live robots.txt after CMS updates because plugins often reintroduce blanket AI disallows.
Decision checklist for Google-Extended
- Confirm Google-Extended appears in your live robots.txt user-agent sections.
- Separate policy goals: Search visibility (Googlebot) vs generative AI fetch (Google-Extended).
- Check legal or compliance guidance for your content type.
- If allowing, verify no CDN rule still blocks Google crawlers at the edge.
- Revisit quarterly as Google documentation evolves.
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report on how robots.txt treats Googlebot and Google-Extended on your domain.
Frequently asked questions
Will allowing Google-Extended improve AI Overviews?
No guarantee. AI Overviews draw heavily from the Search index; Google-Extended is a separate crawl control.
Is blocking Google-Extended enough to opt out of AI training?
It controls that user-agent string on your site. Other policies, products, and crawlers may still apply — verify official Google guidance.
Can I allow Googlebot but block Google-Extended?
Yes. That is a common pattern when you want Search indexing without generative AI fetch from that crawler.
Related questions
Updated
