Hello all,

There must be something that i miss when trying to setup rate limiting on our 0.7.67 installation

We are using nginx as a load-balancer for a rails application, and recently we have been hit by a very badly written bot that generates up to 10 req/s without checking the results. Much like a e-mail harvesting bot, very badly written ...

The first thing i did was to make sure their requests weren't forwarded to the backend, issuing a 444 and hoping that they would notice and just stop doing what they are unsuccessfully trying to do :-)

That didn't stop them. So currently, i am blacklisting their IPs each time i see that bot reappearing. Guess what ... they are on EC2 instances and seem to be very flexible when it comes to changing their IP address :-)

I grew a bit tired after twelve attempts, so i'm looking for alternate possibilities. Note that we have some legitimate traffic on the same server that can reach that amount of requests, so i can't just blindly apply global rate limiting or add rules to the firewall.

Their IP can change, they are using a very specific user-agent, and are hitting a very specific location that isn't normally available on this server, so i came up with the following config:

[code]
http {
include /opt/nginx/conf/mime.types;
default_type application/octet-stream;

geoip_country /opt/nginx/conf/GeoIP.dat;
geoip_city /opt/nginx/conf/GeoLiteCity.dat;

limit_req_zone $binary_remote_addr zone=one:10m rate=2r/s;
limit_req_zone $http_user_agent zone=blackhole:10m rate=1r/m;
limit_zone two $binary_remote_addr 10m;
...
server {
...
location ^~ /api/ {
limit_req zone=blackhole;
limit_conn two 1;
return 444;
}
...
[/code]


But it doesn't seem to work, my server is still hit with many req/s, much more than what i specified in the configuration.

What am i doing wrong here ? do limit_* directives really work in "location" directives ?

What i ultimately would like to do is implement some kind of blackhole for those guys:
- make their request sit there idling for a long time
- make sure they don't take too much simultaneous connections, so we have workers available for legit requests (limit_conn

Is there such a module available ?

Thanks a lot in advance,

Erwan