On Mon, Sep 26, 2016 at 07:41:12PM -0400, c0nw0nk wrote:

Hi there,

> Whats a good setting that won't effect legitimate decent (I think I just
> committed a crime calling some of these companies decent?) crawlers like
> Google, Bing, Baidu, Yandex etc.

Look at your logs for traffic from Google, Bing, Baidu, Yandex etc. How
many of these requests do they make per second, when they make requests?

Set your limits to allow those.

Note that any of the clients can change their crawl-rate on your site at
any time, so any number that worked for you yesterday will not necessarily
work tomorrow.

Cheers,

f
--
Francis Daly francis@daoine.org

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx