After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
By increasing that to a much larger number I haven't noticed slow
accepts/response headers. Also for reference, backlog on a listen
socket is silently limited to net.core.somaxconn (which defaults to
128) so make sure to increase that and other necessary tunings as well.
On Wed, Oct 10, 2012 at 01:11:42PM -0700, John Watson <john@disqus.com> wrote:
> 1) Error logs are clean (except for some 404s)
>
> 2) nginx.conf and sysctl.conf: https://gist.github.com/0b3b52050254e273ff11
>
> Set TX/RX descriptors to 4096/4096 (maximum):
> ethtool -G eth1 tx 4096 rx 4096
>
> Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC
>
> Don't know exact amount, but a good majority of the connections are
> sitting idle for 90s before being closed.
>
> Some graphs on the network interface for past couple days:
> https://www.dropbox.com/s/0bl304ulhqp6a4n/push_stream_network.png
>
> Thank you,
>
> John W
>
> On Wed, Oct 10, 2012 at 01:05:05PM +0400, Andrew Alexeev <andrew@nginx.com> wrote:
> > John,
> >
> > On Oct 9, 2012, at 11:10 PM, John Watson wrote:
> >
> > > I was wondering if anyone had some tips/guidelines for scaling Nginx on
> > > Linux to >500k concurrent connections. Playing with the
> > > nginx_http_push_stream module in streaming mode. Noticing periodic slow
> > > accept and/or response headers. I've scoured the Internet
> > > looking/learning ways to tune Nginx/Linux but I think I've exhausted my
> > > abilities.
> > >
> > > Any help would be appreciated.
> > >
> > > Hardware
> > > Dual Nehalem 5520
> > > 24G RAM
> > > Intel 82576 (igb)
> > > Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
> > >
> > > Thank You,
> > >
> > > John W
> >
> > I'd assume you've already checked/fixed the following, right?
> >
> > 1) Error logs - anything wrong seen in there?
> >
> > 2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try it on/off?
> >
> > 3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker_rlimit_nofile)
> >
> > 4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all those connections to upstreams)
> >
> > Additional information about what's happening in all those 500k connections might be helpful, as well as the relevant configuration section :)
> >
> > Hope this helps
> >
> >
> > --
> > AA @ nginx
> > http://nginx.com/support.html
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
