Home › Knowledge base › AI crawlers in detail

Google-Extended and Googlebot

Google-Extended is not a separate crawler but a robots.txt token you use to control whether your content is used for Gemini training.

Two separate levers

The fetch for Google Search and AI Overviews runs through the regular Googlebot. Google-Extended only decides whether your content may additionally be used to train generative Google models.

Excluding training only

"User-agent: Google-Extended" followed by "Disallow: /" removes you from Gemini training without affecting visibility in Google Search or AI Overviews — Googlebot stays allowed.

What this does not do

Google-Extended does not prevent a snippet of your page from being cited in an AI Overview — that follows normal search indexing. To avoid appearing there, you would have to restrict the page for Googlebot and lose classic search too.

Checking the status

citeglass reads your robots.txt and shows the applicable rule for each bot — Googlebot and Google-Extended — including whether it applies via a wildcard or a dedicated block.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Google AI Overviews
• Allow or block GPTBot
• Block training, allow search

General information, not legal advice. GEO is a young field — recommendations may change.