Hi community,

We are using Nginx as static http server to cache/send static smal file.
The average file size is about 10k ~ 200k -, and nginx server can process
~10000 request per second. It runs well but some times most of nginx worker
go into Stat D. and there's no way to kill them but restart system.

Here's the call stack for kernel/user space:

With AIO enabled
-----Kernel Space Call Stack-----
0xffffffff8111f700 : sync_page+0x0/0x50 [kernel]
0xffffffff8111f75e : sync_page_killable+0xe/0x40 [kernel]
0xffffffff81529e7a : __wait_on_bit_lock+0x5a/0xc0 [kernel]
0xffffffff8111f667 : __lock_page_killable+0x67/0x70 [kernel]
0xffffffff81121394 : generic_file_aio_read+0x4b4/0x700 [kernel]
0xffffffff811d58d4 : aio_rw_vect_retry+0x84/0x200 [kernel]
0xffffffff811d7294 : aio_run_iocb+0x64/0x170 [kernel]
0xffffffff811d86c1 : do_io_submit+0x291/0x920 [kernel]
0xffffffff811d8d60 : sys_io_submit+0x10/0x20 [kernel]
0xffffffff8100b288 : tracesys+0xd9/0xde [kernel]
-----User Space Call Stack-----
0x3c362e50c9 : syscall+0x19/0x40 [/lib64/libc-2.12.so]
0x4d2232 : ngx_linux_sendfile_chain+0xc2a/0xc2c
[/opt/soft/nginx/sbin/nginx]
0x4d24ea : ngx_file_aio_read+0x2b6/0x528 [/opt/soft/nginx/sbin/nginx]
0x515247 : ngx_http_file_cache_open+0xbef/0x1437
[/opt/soft/nginx/sbin/nginx]
0x514df6 : ngx_http_file_cache_open+0x79e/0x1437
[/opt/soft/nginx/sbin/nginx]
0x514abb : ngx_http_file_cache_open+0x463/0x1437
[/opt/soft/nginx/sbin/nginx]
0x5033d0 : ngx_http_upstream_init+0xbc5/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x502916 : ngx_http_upstream_init+0x10b/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x5028ad : ngx_http_upstream_init+0xa2/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x4f8313 : ngx_http_read_client_request_body+0x117/0xd1c
[/opt/soft/nginx/sbin/nginx]
0x5351da : ngx_http_ssi_map_uri_to_path+0x1aff0/0x3a5e0
[/opt/soft/nginx/sbin/nginx]
0x4df55b : ngx_http_core_content_phase+0x41/0x1c9
[/opt/soft/nginx/sbin/nginx]
0x4de484 : ngx_http_core_run_phases+0x87/0xc2 [/opt/soft/nginx/sbin/nginx]
0x4de3fb : ngx_http_handler+0x1c3/0x1c5 [/opt/soft/nginx/sbin/nginx]
0x4ec468 : ngx_http_process_request+0x304/0xa98
[/opt/soft/nginx/sbin/nginx]
0x4eaed8 : ngx_http_process_request_uri+0x95e/0x1876
[/opt/soft/nginx/sbin/nginx]
0x4ea414 : ngx_http_ssl_servername+0x6d5/0x83b [/opt/soft/nginx/sbin/nginx]
0x4e94ea : ngx_http_init_connection+0x785/0x78f
[/opt/soft/nginx/sbin/nginx]
0x4d11b3 : ngx_os_specific_status+0xe55/0x12aa [/opt/soft/nginx/sbin/nginx]
0x4c3e5a : ngx_process_events_and_timers+0xd6/0x165
[/opt/soft/nginx/sbin/nginx]


Without AIO enabled:

-----Kernel Space Call Stack-----
0xffffffff8111f700 : sync_page+0x0/0x50 [kernel]
0xffffffff8111f75e : sync_page_killable+0xe/0x40 [kernel]
0xffffffff81529e7a : __wait_on_bit_lock+0x5a/0xc0 [kernel]
0xffffffff8111f667 : __lock_page_killable+0x67/0x70 [kernel]
0xffffffff81121394 : generic_file_aio_read+0x4b4/0x700 [kernel]
0xffffffff81188c8a : do_sync_read+0xfa/0x140 [kernel]
0xffffffff81189645 : vfs_read+0xb5/0x1a0 [kernel]
0xffffffff81189972 : sys_pread64+0x82/0xa0 [kernel]
0xffffffff8100b288 : tracesys+0xd9/0xde [kernel]
-----User Space Call Stack-----
0x3c36e0f043 : __pread_nocancel+0xa/0x67 [/lib64/libpthread-2.12.so]
0x4c9ab1 : ngx_read_file+0x35/0xb9 [/opt/soft/nginx/sbin/nginx]
0x5152e9 : ngx_http_file_cache_open+0xc91/0x1437
[/opt/soft/nginx/sbin/nginx]
0x514df6 : ngx_http_file_cache_open+0x79e/0x1437
[/opt/soft/nginx/sbin/nginx]
0x514abb : ngx_http_file_cache_open+0x463/0x1437
[/opt/soft/nginx/sbin/nginx]
0x5033d0 : ngx_http_upstream_init+0xbc5/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x502916 : ngx_http_upstream_init+0x10b/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x5028ad : ngx_http_upstream_init+0xa2/0x7eb4 [/opt/soft/nginx/sbin/nginx]
0x4f8313 : ngx_http_read_client_request_body+0x117/0xd1c
[/opt/soft/nginx/sbin/nginx]
0x5351da : ngx_http_ssi_map_uri_to_path+0x1aff0/0x3a5e0
[/opt/soft/nginx/sbin/nginx]
0x4df55b : ngx_http_core_content_phase+0x41/0x1c9
[/opt/soft/nginx/sbin/nginx]
0x4de484 : ngx_http_core_run_phases+0x87/0xc2 [/opt/soft/nginx/sbin/nginx]
0x4de3fb : ngx_http_handler+0x1c3/0x1c5 [/opt/soft/nginx/sbin/nginx]
0x4ec468 : ngx_http_process_request+0x304/0xa98
[/opt/soft/nginx/sbin/nginx]
0x4eaed8 : ngx_http_process_request_uri+0x95e/0x1876
[/opt/soft/nginx/sbin/nginx]
0x4ea414 : ngx_http_ssl_servername+0x6d5/0x83b [/opt/soft/nginx/sbin/nginx]
0x4e94ea : ngx_http_init_connection+0x785/0x78f
[/opt/soft/nginx/sbin/nginx]
0x4d11b3 : ngx_os_specific_status+0xe55/0x12aa [/opt/soft/nginx/sbin/nginx]
0x4c3e5a : ngx_process_events_and_timers+0xd6/0x165
[/opt/soft/nginx/sbin/nginx]
0x4cf1a3 : ngx_single_process_cycle+0x1053/0x2114
[/opt/soft/nginx/sbin/nginx]

Is anyone see these issue (pending on Stat D) before? how can I resolve it?
it seems that not the kernel issue.


I have done some basic statistics in 3 seconds:
Kernel function: call times
sync_page: 91
sync_buffer:2
generic_file_aio_read: 3240
sys_read: 3240
sys_write: 698


Thanks
Qiang
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx