AI search fix
Google-Extended vs Googlebot: what is the difference?
Googlebot is Google's crawler for traditional Search indexing — organic rankings, Discover and AI Overviews that source from the Search index. Google-Extended is a distinct user-agent for generative AI and model-related crawls, including Gemini-oriented fetches. Blocking Google-Extended in robots.txt does not stop Googlebot crawling for ordinary search, and allowing Googlebot does not automatically permit Google-Extended. Write explicit rules for each string. Neither choice guarantees citations in Gemini or placement in AI Overviews.
CMS robots editors often add a Google-Extended section only after press coverage of AI training. Record your decision: permit both for the widest AI surface, block Google-Extended whilst keeping Googlebot, or seek legal advice before any blanket AI disallow.
| User-agent | Primary role |
|---|---|
| Googlebot | Classic Google Search crawl and index — blue links and much AI Overviews sourcing. |
| Google-Extended | Generative AI and training-oriented crawl — its own robots.txt section. |
| GEO overlap | Both need reachable HTML, honest content and schema where appropriate. |
| Not the same as | GPTBot, ClaudeBot — third-party AI crawlers use separate strings. |
You'll get an HTML report showing how robots.txt treats Googlebot, Google-Extended and other AI agents.
Run the diagnosticRelated questions
- Google-Extended blocked — should you allow it?Weighing policy options when Google-Extended is currently disallowed.
- robots.txt for AI crawlers — how to write itListing Google-Extended alongside GPTBot, ClaudeBot and others.
- ChatGPT SEO vs Google SEO: key differencesHow channels differ beyond Google's two crawler strings.
Updated