AI search fix
Google-Extended blocked — should you allow it?
Google-Extended is a separate user agent from Googlebot. Google deploys it for generative AI and model-training crawls, not for classic blue-link search indexing. Blocking Google-Extended does not halt normal Google Search crawling, but it can restrict how Google's AI products use your content. Permit it when you want broader Gemini-related access; block it when your policy limits AI training crawls whilst Googlebot remains allowed.
Sites often paste a blanket `Disallow: /` under Google-Extended without distinguishing it from Googlebot. That may suit your legal stance, but consider the trade-off: AI-surface visibility may shrink whilst traditional SEO stays intact. Record the decision and align robots.txt with marketing and compliance teams.
How to set Google-Extended policy in robots.txt
- Fetch your live /robots.txt and find any Google-Extended section.
- Choose allow or block based on AI training and Gemini access policy.
- Keep Googlebot rules in a separate block — avoid blocking search by mistake.
- Deploy and verify the production file, not a staging copy.
- Re-check after SEO plugins or CMS robots editors apply updates.
You'll get an HTML report showing how robots.txt treats Google-Extended, GPTBot and other AI agents.
Run the diagnosticRelated questions
- robots.txt for AI crawlers — how to write itTrusted AI user-agent sections, including Google-Extended.
- AI crawlers blocked by robots.txt — how to fix itRemove policy-level Disallow rules that still block bots.
- Schema markup missing for AI search — how to fix itAdd JSON-LD so pages are easier to parse and cite.
Updated