You can block some of those bots at the firewall permanently.  

I use the nginx map feature in a similar manner, but I don't know if map is more efficient than your code. ‎I started out blocking similar to your scheme, but the map feature looks clear to me in the conf file.

Majestic and Sogou sure are annoying. For what I block, I use 444 rather than 403. (And yes, I know that destroys the matter/anti-matter mix of the universe, so don't lecture me.) I then eyeball the 444 hits periodically, using a script to pull the 444 requests out of the access.log file. I have another script to get just the IP addresses from access.log.

For the search engines like Majestic and Sogou, which don't seem to have an IP space you can look up via BGP tools, I take the IP used and add it to my firewall blocking table. I can go weeks before a new IP gets used.

  Original Message  
From: debilish99
Sent: Monday, November 14, 2016 7:04 AM
To: nginx@nginx.org
Reply To: nginx@nginx.org
Subject: Bloking Bad bots

Hello,

I have a server with several domains, in the configuration file of each
domain I have a line like this to block bad bots.

If ($ http_user_agent ~ *
(zealbot|MJ12bot|AhrefsBot|sogou|PaperLiBot|uipbot|DotBot|GetIntent|Cliqzbot|YandexBot|Nutch|TurnitinBot|IndeedBot)
Return 403;
}

This works fine.

The question is, if I increase the list of bad bots to 1000, for example,
this would be a speed problem when nginx manages every request that
arrives.

I have domains that can have 500,000 hits daily and up to 20,000 hits.

Thank you all.

Greetings.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270930,270930#msg-270930

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx