Home › Knowledge base › AI crawlers in detail

ClaudeBot and the Anthropic crawlers

Anthropic runs three crawlers for Claude: one for training, one for search, and one for fetching on user request.

The three bots

ClaudeBot collects training data. Claude-SearchBot builds the index for web search in Claude. Claude-User fetches a page when a user asks for it in a conversation. Only ClaudeBot concerns model training.

Managing access

One robots.txt block per bot. If you reject training but want to be cited in Claude answers, block ClaudeBot with "Disallow: /" and keep allowing Claude-SearchBot and Claude-User.

A common mistake

A single block "User-agent: Claude*" does not work — robots.txt knows no wildcards in the user-agent name other than the standalone "*". Name each bot individually.

Checking the server response

citeglass fetches your page with the real user agents of ClaudeBot, Claude-SearchBot and Claude-User and flags cases where robots.txt allows a bot but the server rejects it.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Allow or block GPTBot
• PerplexityBot and Perplexity-User
• Controlling AI crawlers in robots.txt

General information, not legal advice. GEO is a young field — recommendations may change.