Home › Knowledge base › AI crawlers in detail

Allow or block GPTBot

GPTBot is the crawler OpenAI uses to collect publicly accessible content for training future models.

What GPTBot is for

GPTBot fetches pages for training data — not to answer a specific user question in ChatGPT. The live search in ChatGPT is handled by OAI-SearchBot and ChatGPT-User. If you block GPTBot but still want to appear in AI search, you must keep allowing those two.

Allowing it

A dedicated block gives GPTBot full access: "User-agent: GPTBot" followed by "Allow: /". Important: a bot-specific block fully replaces the "User-agent: *" block for GPTBot — repeat any necessary Disallow rules there.

Blocking it

"User-agent: GPTBot" followed by "Disallow: /" keeps GPTBot off the whole domain. OpenAI usually respects robots.txt within 24 hours. It does not pull back content that has already gone into training.

Checking whether it works

GPTBot obeys robots.txt — but still won't get through if a WAF or bot protection rejects the user agent with a 403 first. citeglass fetches your page with the real GPTBot user agent and shows whether the robots.txt rule and the actual server response match.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• OAI-SearchBot and ChatGPT-User
• Controlling AI crawlers in robots.txt
• Block training, allow search

General information, not legal advice. GEO is a young field — recommendations may change.