Francis Daly Wrote:
-------------------------------------------------------
> On Mon, Sep 26, 2016 at 07:41:12PM -0400, c0nw0nk wrote:
>
> Hi there,
>
> > Whats a good setting that won't effect legitimate decent (I think I
> just
> > committed a crime calling some of these companies decent?) crawlers
> like
> > Google, Bing, Baidu, Yandex etc.
>
> Look at your logs for traffic from Google, Bing, Baidu, Yandex etc.
> How
> many of these requests do they make per second, when they make
> requests?
>
> Set your limits to allow those.
>
> Note that any of the clients can change their crawl-rate on your site
> at
> any time, so any number that worked for you yesterday will not
> necessarily
> work tomorrow.
>
> Cheers,
>
> f
> --
> Francis Daly francis@daoine.org
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu . Wrote:
-------------------------------------------------------
> Perhaps this can help:
>
> http://stackoverflow.com/questions/12022429/http-status-code-for-overl
> oaded-server
>
> Another option is a *429 - Too Many Requests* response.
>
> Defined in RFC6585 - http://tools.ietf.org/html/rfc6585#section-4
>
> The spec does not define how the origin server identifies the user,
> nor how
> it counts requests.
>
> For example, an origin server that is limiting request rates can do so
> based upon counts of requests on a per-resource basis, across the
> entire
> server, or even among a set of servers.
>
> Likewise, it might identify the user by its authentication
> credentials, or
> a stateful cookie.
>
> Also see the Retry-After header in the response.
>
> Kind regards,
> Oscar
>
>
> On Tue, Sep 27, 2016 at 10:59 AM, itpp2012
> <nginx-forum@forum.nginx.org>
> wrote:
>
> > General rule of thumb is set it as low as possible, as soon as 503's
> are
> > getting your users upset or resources are getting blocked, then
> double the
> > values, keep an eye on the logs, double it one more time when
> required,
> > done.
> >
> > Posted at Nginx Forum: https://forum.nginx.org/read.
> > php?2,269862,269866#msg-269866
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
>
>
>
> --
> Oscar Fernandez Sierra
> oscaretu@gmail.com
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


Thanks raises a question of why the default status code on limit_conn and limit_req is a 503 instead of a 429.

limit_conn_status 503;
limit_req_status 503;

But under a DoS attack I always feel those values would be better being "444" since the server won't respond and cut's the connection rather than waste bandwidth on a client who is opening and closing connections fast as a bullet. You tend to get allot of 499's in your access.log when really you should not even waste your bandwidth output trying to respond to them.

Is it worth me changing the default status codes or keeping them at default.

Also I am going to start out with itpp2012's advice setting it at the low value and increasing it based on what companies crawlers require a more brutal crawl rate Francis Daly's idea is also good though but since I use "burst=5" instead of "nodelay" on my request limit I get to find out what crawlers would actually wait for the current request that they are experiencing a loading page on to finish.

http://www.networkflare.com/