1) Error logs are clean (except for some 404s)
2) nginx.conf and sysctl.conf: https://gist.github.com/0b3b52050254e273ff11
Set TX/RX descriptors to 4096/4096 (maximum):
ethtool -G eth1 tx 4096 rx 4096
Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC
Don't know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.
Some graphs on the network interface for past couple days:
https://www.dropbox.com/s/0bl304ulhqp6a4n/push_stream_network.png
Thank you,
John W
On Wed, Oct 10, 2012 at 01:05:05PM +0400, Andrew Alexeev <andrew@nginx.com> wrote:
> John,
>
> On Oct 9, 2012, at 11:10 PM, John Watson wrote:
>
> > I was wondering if anyone had some tips/guidelines for scaling Nginx on
> > Linux to >500k concurrent connections. Playing with the
> > nginx_http_push_stream module in streaming mode. Noticing periodic slow
> > accept and/or response headers. I've scoured the Internet
> > looking/learning ways to tune Nginx/Linux but I think I've exhausted my
> > abilities.
> >
> > Any help would be appreciated.
> >
> > Hardware
> > Dual Nehalem 5520
> > 24G RAM
> > Intel 82576 (igb)
> > Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
> >
> > Thank You,
> >
> > John W
>
> I'd assume you've already checked/fixed the following, right?
>
> 1) Error logs - anything wrong seen in there?
>
> 2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try it on/off?
>
> 3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker_rlimit_nofile)
>
> 4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all those connections to upstreams)
>
> Additional information about what's happening in all those 500k connections might be helpful, as well as the relevant configuration section :)
>
> Hope this helps
>
>
> --
> AA @ nginx
> http://nginx.com/support.html
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
