Home › Knowledge base › AI crawlers in detail

Meta-ExternalAgent

Meta-ExternalAgent is the crawler Meta uses to collect publicly accessible content for Llama models and Meta AI.

Names and purpose

Meta also uses the older name "FacebookBot" and "meta-externalfetcher" for fetches on user request. Meta-ExternalAgent is the bot relevant for training and AI features.

Blocking it

"User-agent: Meta-ExternalAgent" followed by "Disallow: /". Additionally "User-agent: meta-externalagent" in lowercase, since the spelling varies in practice.

The trade-off

A block removes you from Llama training and from Meta AI answers in WhatsApp, Instagram and Facebook. For local businesses with social reach, visibility here can be valuable.

Checking access

citeglass fetches your page with the Meta-ExternalAgent user agent and shows whether robots.txt and the server response match.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• Bytespider by ByteDance
• Block all AI crawlers
• Controlling AI crawlers in robots.txt

General information, not legal advice. GEO is a young field — recommendations may change.