Home › Knowledge base › Topics

Block all AI crawlers

Anyone who wants to keep content out of AI systems on principle needs rules for every named bot — a wildcard is not enough, because many bots ignore it or expect their own blocks.

The list of bots

At minimum, you need to account for: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, Meta-ExternalAgent. One dedicated block per bot with "Disallow: /".

What robots.txt cannot do

robots.txt is a request, not enforcement. Bots like Bytespider or Perplexity have been criticised for bypassing it. For hard blocks, add WAF or firewall rules on the user agents and, where possible, on verified IP ranges.

The price

A complete block removes you from AI answers with a source link — that is, from a growing part of information seeking. For many sites the more targeted variant makes more sense: block training, allow search.

Checking the effect

citeglass fetches your page with each of these user agents and shows in a matrix which bot is blocked per robots.txt and whether your server actually enforces the block.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Block training, allow search
• Controlling AI crawlers in robots.txt
• CCBot and Common Crawl

General information, not legal advice. GEO is a young field — recommendations may change.