Magento – excessive traffic from meta-externalagent

You can rate-limit this bot using nginx built-in rate limiting functionality. According to the "Meta Web Crawlers" Facebook artice, the User-Agent HTTP header for this bot can be either

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

or

meta-externalagent/1.1

However, since the article may not list all possible User-Agent values, you should check your nginx access logs to be sure.

To limit requests from this bot, you can add the following snippet to your nginx configuration:

map $http_user_agent $meta_crawler {
    "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"  1;
    "meta-externalagent/1.1"                                                                     1;
}
# Limit meta crawler to 30 requests per minute; adjust according to your needs
limit_req_zone $meta_crawler zone=meta:10m rate=30r/m;

server {
    # your magento server section
    limit_req zone=meta burst=5 nodelay;
    limit_req_status 429;
    ... # the rest of your nginx configuration
}

It is recommended to include the Retry-After HTTP header in the HTTP 429 response; you can do it as follows:

server {
    limit_req zone=meta burst=5 nodelay;
    limit_req_status 429;
    error_page 429 @err429;
    location @err429 {
        add_header Retry-After 30 always;
        return 429;
    }
    ... # the rest of your nginx configuration
}

Although you can use regular expressions in the map block, e.g.:

map $http_user_agent $meta_crawler {
    "~^meta-externalagent/"  1;
}

I do not recommend doing it due to the performance considerations (read this nginx support forum post for the details).