Meta-ExternalAgent: what it is and how to block it
Crawls the web for uses such as training foundation AI models or indexing content for Meta products.
Does Meta-ExternalAgent follow robots.txt?
Meta shows a robots.txt example for meta-externalagent and says changes can take up to 24 hours because crawlers cache robots.txt.
What blocking it does
Opts your site out of this crawler, including its AI-training use.
Other Meta tokens
- Meta-ExternalFetcher: user-requested fetch. Fetches individual links at a user’s request, including for agentic AI features.
Block Meta-ExternalAgent everywhere
User-agent: Meta-ExternalAgent
Disallow: /
Block only some folders
User-agent: Meta-ExternalAgent
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Meta-ExternalAgent
Allow: /
A group that names Meta-ExternalAgent replaces the * group for it. Copy any shared rules into this group too. Test the result.
Meta-ExternalAgent questions
What is Meta-ExternalAgent?
Meta-ExternalAgent is operated by Meta. Crawls the web for uses such as training foundation AI models or indexing content for Meta products.
How do I block Meta-ExternalAgent?
Add "User-agent: Meta-ExternalAgent" followed by "Disallow: /" to your robots.txt. Meta shows a robots.txt example for meta-externalagent and says changes can take up to 24 hours because crawlers cache robots.txt.
What happens if I block Meta-ExternalAgent?
Opts your site out of this crawler, including its AI-training use.
Does blocking Meta-ExternalAgent block other Meta crawlers?
No. Meta documents Meta-ExternalFetcher separately; each needs its own User-agent line if you want to block it.