Hello!

On Fri, Sep 26, 2014 at 03:45:40PM +0800, Zhang Qiang wrote:

> Hi community,
>
> We are using Nginx as static http server to cache/send static smal file.
> The average file size is about 10k ~ 200k -, and nginx server can process
> ~10000 request per second. It runs well but some times most of nginx worker
> go into Stat D. and there's no way to kill them but restart system.
>
> Here's the call stack for kernel/user space:
>
> With AIO enabled
> -----Kernel Space Call Stack-----
> 0xffffffff8111f700 : sync_page+0x0/0x50 [kernel]
> 0xffffffff8111f75e : sync_page_killable+0xe/0x40 [kernel]
> 0xffffffff81529e7a : __wait_on_bit_lock+0x5a/0xc0 [kernel]
> 0xffffffff8111f667 : __lock_page_killable+0x67/0x70 [kernel]
> 0xffffffff81121394 : generic_file_aio_read+0x4b4/0x700 [kernel]
> 0xffffffff811d58d4 : aio_rw_vect_retry+0x84/0x200 [kernel]
> 0xffffffff811d7294 : aio_run_iocb+0x64/0x170 [kernel]
> 0xffffffff811d86c1 : do_io_submit+0x291/0x920 [kernel]
> 0xffffffff811d8d60 : sys_io_submit+0x10/0x20 [kernel]
> 0xffffffff8100b288 : tracesys+0xd9/0xde [kernel]
> -----User Space Call Stack-----
> 0x3c362e50c9 : syscall+0x19/0x40 [/lib64/libc-2.12.so]
> 0x4d2232 : ngx_linux_sendfile_chain+0xc2a/0xc2c
> [/opt/soft/nginx/sbin/nginx]
> 0x4d24ea : ngx_file_aio_read+0x2b6/0x528 [/opt/soft/nginx/sbin/nginx]
> 0x515247 : ngx_http_file_cache_open+0xbef/0x1437
> [/opt/soft/nginx/sbin/nginx]
> 0x514df6 : ngx_http_file_cache_open+0x79e/0x1437
> [/opt/soft/nginx/sbin/nginx]
> 0x514abb : ngx_http_file_cache_open+0x463/0x1437
> [/opt/soft/nginx/sbin/nginx]

Just a side note: it looks like there are very wrong symbols shown
in the stack, likely due to optimizations used.

[...]

> Is anyone see these issue (pending on Stat D) before? how can I resolve it?
> it seems that not the kernel issue.

What makes you think so? It looks like the kernel issue from here -
"unkillable" proceses just can't happen due to userland code
unless there is a problem in the kernel.

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx