AI search fix
Google-Extended vs Googlebot: what is the difference?
Googlebot is Google's crawler for traditional Search indexing — rankings, Discover, and AI Overviews that draw from the Search index. Google-Extended is a separate user-agent Google uses for generative AI and model-related crawls, including Gemini-oriented access. Blocking Google-Extended in robots.txt does not stop Googlebot from crawling for normal search, and allowing Googlebot does not automatically allow Google-Extended. Publish explicit rules for each user-agent. Neither policy guarantees citations in Gemini or placement in AI Overviews.
Many CMS robots editors add Google-Extended only after a headline about AI training. Document your policy: allow both for maximum AI surface access, block Google-Extended only while keeping Googlebot, or consult legal before any blanket AI disallow.
| User-agent | Primary role |
|---|---|
| Googlebot | Classic Google Search crawl and index — blue links, much of AI Overviews sourcing. |
| Google-Extended | Generative AI / training-oriented crawl — separate robots.txt control. |
| GEO overlap | Both need reachable HTML, honest content, and schema where appropriate. |
| Not the same as | GPTBot, ClaudeBot — third-party AI crawlers use their own strings. |
Free · 2 minutes · no card
See what AI crawlers hit on your site
Technical blockers, missing context, weak AI-readiness signals — in one HTML report.
You'll get an HTML report showing how robots.txt treats Googlebot, Google-Extended, and other AI agents.
Frequently asked questions
If I block Google-Extended, do I lose Google SEO?
Traditional ranking crawls use Googlebot. Blocking Google-Extended targets Google's separate generative AI crawler — not a replacement for blocking Googlebot. Always verify both sections in robots.txt.
Should I allow Google-Extended for GEO?
Allow it if your policy permits Google's generative AI systems to fetch public pages. Block it if you restrict AI training crawls while keeping search visibility via Googlebot.
Does Google-Extended replace llms.txt for Google?
No. Google has stated llms.txt is not required for Search generative features when crawl and helpful content are solid. Google-Extended is a robots.txt user-agent, not a file at /llms.txt.
Related questions
Updated