Welcome! Log In Create A New Profile

Advanced

Re: Bloking Bad bots

November 14, 2016 10:32AM
You can block some of those bots at the firewall permanently.  

I use the nginx map feature in a similar manner, but I don't know if map is more efficient than your code. ‎I started out blocking similar to your scheme, but the map feature looks clear to me in the conf file.

Majestic and Sogou sure are annoying. For what I block, I use 444 rather than 403. (And yes, I know that destroys the matter/anti-matter mix of the universe, so don't lecture me.) I then eyeball the 444 hits periodically, using a script to pull the 444 requests out of the access.log file. I have another script to get just the IP addresses from access.log.

For the search engines like Majestic and Sogou, which don't seem to have an IP space you can look up via BGP tools, I take the IP used and add it to my firewall blocking table. I can go weeks before a new IP gets used.

  Original Message  
From: debilish99
Sent: Monday, November 14, 2016 7:04 AM
To: nginx@nginx.org
Reply To: nginx@nginx.org
Subject: Bloking Bad bots

Hello,

I have a server with several domains, in the configuration file of each
domain I have a line like this to block bad bots.

If ($ http_user_agent ~ *
(zealbot|MJ12bot|AhrefsBot|sogou|PaperLiBot|uipbot|DotBot|GetIntent|Cliqzbot|YandexBot|Nutch|TurnitinBot|IndeedBot)
Return 403;
}

This works fine.

The question is, if I increase the list of bad bots to 1000, for example,
this would be a speed problem when nginx manages every request that
arrives.

I have domains that can have 500,000 hits daily and up to 20,000 hits.

Thank you all.

Greetings.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270930,270930#msg-270930

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Bloking Bad bots

debilish99 November 14, 2016 10:04AM

Re: Bloking Bad bots

gariac November 14, 2016 10:32AM

Re: Bloking Bad bots

Anoop Alias November 14, 2016 10:42AM

Re: Bloking Bad bots

gariac November 14, 2016 11:52AM

Re: Bloking Bad bots

Robert Paprocki November 14, 2016 11:58AM

Re: Bloking Bad bots

Anonymous User November 14, 2016 12:08PM

Re: Bloking Bad bots

ph.gras November 14, 2016 12:14PM

Re: Bloking Bad bots

George November 14, 2016 05:18PM

Re: Bloking Bad bots

Star Dot November 15, 2016 12:24AM

Re: Bloking Bad bots

gariac November 15, 2016 03:56AM

Re: Bloking Bad bots

debilish99 November 15, 2016 04:05AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 112
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready