Home › Knowledge base › Basics

Controlling AI crawlers in robots.txt

Every major AI provider runs its own crawlers with their own user-agent names. Via robots.txt you decide, per bot, whether it may load your content.

The main AI crawlers

OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetch on user request). Anthropic: ClaudeBot, Claude-SearchBot, Claude-User. Perplexity: PerplexityBot, Perplexity-User. Google: Google-Extended (Gemini training) — the actual fetch for AI Overviews runs via the normal Googlebot. Plus Common Crawl (CCBot), Amazonbot, Bytespider, Applebot-Extended and Meta-ExternalAgent.

Allowing specifically

For maximum AI visibility, give the relevant bots their own block with "Allow: /". Important: a bot-specific block fully replaces the "User-agent: *" block for that bot — repeat any needed disallow rules there.

Excluding specifically

If you reject training but want to stay visible in AI search, you can block GPTBot and Google-Extended while allowing OAI-SearchBot and Googlebot. That's a content decision, not a purely technical one.

Why robots.txt is often not enough

Many sites allow bots in robots.txt but block them at the server level: a WAF rule, a Cloudflare "Bot Fight Mode" or a firewall turns the crawler away with a 403. The bot obeys robots.txt and still can't get in. citeglass makes this discrepancy visible.

Verifying

Allow lists should contain verified bots. OpenAI, Anthropic, Perplexity and Google publish the IP ranges of their crawlers; additionally check via reverse DNS instead of relying only on the user-agent string.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

General information, not legal advice. GEO is a young field — recommendations may change.