Home › Knowledge base › AI crawlers in detail

CCBot and Common Crawl

Common Crawl creates a freely available web archive that feeds into the training data of numerous language models.

Why CCBot is special

If you block only GPTBot and ClaudeBot, your content can still reach models via the Common Crawl dataset. Anyone who rejects training on principle has to account for CCBot.

Blocking it

"User-agent: CCBot" followed by "Disallow: /". Common Crawl respects robots.txt on the next crawl run; existing archive entries remain until the dataset is re-collected.

The trade-off

Common Crawl is also used for research, search projects and archiving. A block affects all of these purposes, not just commercial AI training.

Checking the rule

citeglass shows whether your robots.txt allows or blocks CCBot and whether the rule is bot-specific or only applies via the "User-agent: *" block.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Allow or block GPTBot
• Block all AI crawlers
• Controlling AI crawlers in robots.txt

General information, not legal advice. GEO is a young field — recommendations may change.