AI search fix
Google-Extended blocked — should you allow it?
Google-Extended is a separate user agent from Googlebot. It is used for Google's generative AI and model-training crawls, not for classic blue-link search indexing. Blocking Google-Extended does not stop normal Google Search crawling, but it can limit how Google's AI products use your content. Allow it if you want broader Gemini-related access; block it if your policy restricts AI training crawls while keeping Googlebot allowed.
Many sites copy a blanket `Disallow: /` for Google-Extended without separating it from Googlebot. That is valid for policy reasons, but measure the trade-off: you may reduce AI-surface visibility while keeping traditional SEO intact. Document the decision and align robots.txt with your legal and marketing stance.
How to set Google-Extended policy in robots.txt
- Open your live /robots.txt and locate any Google-Extended section.
- Decide allow vs block based on AI training and Gemini access policy.
- Keep Googlebot rules separate — do not block Googlebot by mistake.
- Deploy and verify the production file, not a staging copy.
- Re-check after SEO plugins or CMS robots editors run updates.
You'll get an HTML report showing how robots.txt treats Google-Extended, GPTBot, and other AI agents.
Run Express CheckRelated questions
Updated
