Meta-ExternalFetcher: what it is and how to block it
Fetches individual links at a user’s request, including for agentic AI features.
Does Meta-ExternalFetcher follow robots.txt?
Meta says this crawler may bypass robots.txt because the fetches are user-requested.
What blocking it does
A signal only; Meta does not commit to honouring it.
Because Meta describes Meta-ExternalFetcher as fetching pages on a user's behalf, treat a robots.txt rule as a request rather than a guarantee. For a hard block, filter the user agent or Meta's published IP ranges at your server or CDN.
Other Meta tokens
- Meta-ExternalAgent: model training. Crawls the web for uses such as training foundation AI models or indexing content for Meta products.
Block Meta-ExternalFetcher everywhere
User-agent: Meta-ExternalFetcher
Disallow: /
Block only some folders
User-agent: Meta-ExternalFetcher
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Meta-ExternalFetcher
Allow: /
A group that names Meta-ExternalFetcher replaces the * group for it. Copy any shared rules into this group too. Test the result.
Meta-ExternalFetcher questions
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is operated by Meta. Fetches individual links at a user’s request, including for agentic AI features.
How do I block Meta-ExternalFetcher?
Add "User-agent: Meta-ExternalFetcher" followed by "Disallow: /" to your robots.txt. Meta says this crawler may bypass robots.txt because the fetches are user-requested.
What happens if I block Meta-ExternalFetcher?
A signal only; Meta does not commit to honouring it.
Does blocking Meta-ExternalFetcher block other Meta crawlers?
No. Meta documents Meta-ExternalAgent separately; each needs its own User-agent line if you want to block it.