Hi Eric, This is a rather shameless plug here, but I wrote an nginx
module designed to efficiently block (or filter) large numbers of IP
addresses. It's a two part system with the nginx module being
https://github.com/tmthrgd/nginx-ip-blocker and a separate agent daemon
here https://github.com/tmthrgd/ip-blocker-agent . It uses shared memory
to store the IP addresses and binary search to iterate through them. It
might not work for your circumstance, but it just might. Kind Regards,
Tom Thorogood.

On Wed, 2 Nov 2016, at 09:13 AM, Cox, Eric S wrote:
> Unfortunately much like others have stated, we also don't have the
> automation at the firewall layer to move as quickly as we would like.
> So at the moment its not an option.
>
> -----Original Message----- *From:* Rainer Duffner [rainer@ultra-
> secure.de] *Received:* Tuesday, 01 Nov 2016, 6:41PM *To:*
> nginx@nginx.org [nginx@nginx.org] *Subject:* Re: Blocking tens of
> thousands of IP's
>
>
>> Am 01.11.2016 um 23:35 schrieb Cox, Eric S <eric.cox@kroger.com>:
>>
>> Currently we track all access logs realtime via an in house built log
>> aggregation solution. Various algorithms are setup to detect said IPS
>> whether it be by hit rate, country, known types of attacks etc. These
>> IPS are typically identified within a few mins and we reload to
>> banned list every 60 seconds. We just moved some services from apache
>> where we were doing this without any noticable performance impact.
>> Have this working in nginx but was looking for general suggestion on
>> how to optimize if at all possible.
>
>
> Ah, if you already have the data pre-processed…
>
> I’d move blocking to the host’s firewall, as suggested.
>
> Long term, I want to do this (or at least be able to), too.
>
> We (MSP) have a rather large number of firewalls and telling the network-
> guys „Block this IP at all of them“ does not work (it would probably
> take them the better part of the day).
> They don’t believe in automation...
>
>
> This e-mail message, including any attachments, is for the sole use of
> the intended recipient(s) and may contain information that is
> confidential and protected by law from unauthorized disclosure. Any
> unauthorized review, use, disclosure or distribution is prohibited. If
> you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> _________________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx