On Fri, Oct 9, 2009 at 4:40 PM, Jim Ohlstein <jim@ohlste.in> wrote:
> The nginx forum had a DDoS attack which took the site down this morning. In
> approximately 23 seconds there were just under 900,000 lines in the error
> log that looked like:
>
> 2009/10/09 10:21:38 [alert] 32576#0: accept() failed (24: Too many open
> files)
>
> First question is do each of these entries represent an attempted
> connection?
>
>
> Looking at the access log there were thousands of requests for the same page
> from roughly 400 IP's in that same 23 second span like this:
>
> 58.53.85.229 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
> (compatible; MSIE 6.0; Windows 5.1)"
> 60.177.29.231 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
> (compatible; MSIE 6.0; Windows 5.1)"
> 125.91.207.11 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.0
> (compatible; MSIE 6.0; Windows 5.1)"
> 125.119.65.194 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
> HTTP/1.1" 500 553 "http://forum.nginx.org" "Mozilla/4.
> 0 (compatible; MSIE 6.0; Windows 5.1)"
>
>
> All of the IP's that I checked were legitimate IP's that localize to various
> regions in China.
>
> I have set up limit_zone and limit_conn directives to hopefully mitigate
> this in the future.
>
> Second question is where to set limit_conn and what are the effects on users
> if set low? The site generally responds quickly, at least here in the US,
> and I don't want it to be especially sluggish for people using less fast
> connections in other parts of the world, but of course I want to reduce the
> chances of this happening again. Bear in mind this is a low traffic site
> (16K visits in the last month) on a small VPS.
>
> Any advice would be appreciated.
>
> --
> Jim Ohlstein
>
>
Jim,
Your main issue is going ot be the system 'open files' setting, can
you show me the output of the following please, run the command as
room or the user that runs the nginx proc. 'ulimit -a'
by default linux sets this value to which really means that you cant
have more than 1024 file descriptors open on the system at one time.
You can modify this setting to safely allow up to 150,000 on a more
resent dual cor box.
'ulimit -n 150000'
try reading up on ulimit, its a huge for system performance, also make
sure your not running any iptables connection tracking else you will
need to fine tune those settings as well.
--
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer
