Home › Knowledge base › AI crawlers in detail

OAI-SearchBot and ChatGPT-User

When ChatGPT searches the web, OAI-SearchBot and ChatGPT-User fetch pages — separately from the training crawler GPTBot.

Two bots, two jobs

OAI-SearchBot builds the index for ChatGPT search. ChatGPT-User fetches a page when a user explicitly asks for it in their question or shares a link. Both provide the basis for cited answers with a source link.

Why you should allow them

Anyone who wants to appear as a source in ChatGPT must grant these two bots access. Blocking only GPTBot leaves search untouched; blocking "User-agent: *" also hits OAI-SearchBot.

The robots.txt rule

For maximum visibility: one block each for "User-agent: OAI-SearchBot" and "User-agent: ChatGPT-User", each with "Allow: /". Exclude sensitive areas (login, cart, internal search) with targeted Disallow rules.

Testing retrievability

citeglass checks separately for OAI-SearchBot and ChatGPT-User whether robots.txt allows them and whether your server actually serves them content — including the cases where bot protection blocks them despite an allowing robots.txt.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Allow or block GPTBot
• How ChatGPT cites sources
• Controlling AI crawlers in robots.txt

General information, not legal advice. GEO is a young field — recommendations may change.