I am trying to figure out if there is any way to rate limit all traffic
except Googlebot, msnbot, yandex and baidu bots. Here is what I have
started with:


# Whitelisted IPs
geo $rate_limit_ip {
default $binary_remote_addr;
127.0.0.1 "";
10.0.0.0/8 "";
}

# Rate limit
limit_req_zone $rate_limit_ip zone=publix:10m rate=10r/s;

I can add googlebot, msnbot, yandex and baidu IP ranges manually to the
whitelist, but that will make lookup table big. I am not sure whether this
approach will work for high traffic like - 1200 requests/second distributed
across 20 nginx hosts. Any ideas on such setup will be really helpful.

Also, can such host lookups be done in real-time for every request? I am
guessing that may not be efficient for each request, but I was wondering if
there are any solutions.

Appreciate all your help.

- thanks,
N
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx