Hello!

On Sun, Jun 09, 2013 at 07:24:49PM +0430, Host DL wrote:

> Hello Maxim,
>
> Thanks for your response, and sorry that I am new to mailing list and my
> 1st message may was not very clear to you
>
> I've already read all posts in this conversation and all tuning options has
> been tested
>
> I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
> CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5
>
> nginx.conf:
>
> worker_priority -10;
> worker_processes 64;
> worker_rlimit_nofile 20000;
>
> events {
> worker_connections 2048;
> use epoll;
> worker_aio_requests 128;
> }
>
> http {
> sendfile off;
> tcp_nopush on;
> tcp_nodelay on;
> aio on;
> directio 2m;
> #directio_alignment 4k;
> output_buffers 1 1m;
>
> keepalive_timeout 15;
>
> ......
> }
>
> During the peak time connections will reach up to 14-15K in total and more
> than 1Gbit/s outgoing throughput
> Please note that the server was stable with about 12K connections in the
> peak time and about 1-1.1Gbit/s throughput but after adding another VH with
> about 2-3K connections it seems that server is unable to handle the request
> properly at the peak time
> Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
> doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
> getting more and bigger
>
> During every peak time the LA will each to the number of nginx workers ( 64
> for my current config ) and will stay at the same rate to the end of peak
> time, all processes are in D state and the interesting thing, memory is not
> being used fully and it may push about 30-40G with about 20-30% I/O wait

Main problem with AIO on Linux is that it requires directio to
actually work asynchronously. I would assume you've just reached
a critical number of synchronous requests to disks as due to
"directio 2m" in your config (and that's why you see all workers
in D state). Try tuning directio to a lower value to see if it
helps.

Note well: memory is not used for filesystem cache with directio,
so there is no surprise it's not being used fully.

--
Maxim Dounin
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx