AI search fix
Google-Extended vs Googlebot: what is the difference?
Googlebot is Google's crawler for traditional Search indexing — organic rankings, Discover, and AI Overviews that source from the Search index. Google-Extended is a distinct user-agent for generative AI and model-related crawls, including Gemini-oriented fetches. Blocking Google-Extended in robots.txt does not stop Googlebot crawling for ordinary search, and allowing Googlebot does not automatically permit Google-Extended. Write explicit rules for each string. Neither choice guarantees citations in Gemini or placement in AI Overviews.
CMS robots editors often add a Google-Extended section only after press coverage of AI training. Record your decision: permit both for the widest AI surface, block Google-Extended whilst keeping Googlebot, or seek legal advice before any blanket AI disallow.
| User-agent | Primary role |
|---|---|
| Googlebot | Classic Google Search crawl and index — blue links and much AI Overviews sourcing. |
| Google-Extended | Generative AI and training-oriented crawl — its own robots.txt section. |
| GEO overlap | Both need reachable HTML, honest content, and schema where appropriate. |
| Not the same as | GPTBot, ClaudeBot — third-party AI crawlers use separate strings. |
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report showing how robots.txt treats Googlebot, Google-Extended, and other AI agents.
Frequently asked questions
If I block Google-Extended, do I lose Google SEO?
Traditional ranking crawls rely on Googlebot. Blocking Google-Extended targets Google's generative AI crawler — check both sections in live robots.txt.
Should I allow Google-Extended for GEO?
Permit it if your policy allows Google's generative AI systems to fetch public pages. Block it if you restrict AI training crawls whilst keeping search visibility via Googlebot.
Does Google-Extended replace llms.txt for Google?
No. Google has stated llms.txt is not required for Search generative features when crawl and helpful content are solid.
Related questions
Updated