Welcome! Log In Create A New Profile

Advanced

Re: nginx worker process hang,cpu load 100%

November 14, 2011 09:37AM
Hello,Maxim. Thanks for you help.

I reproduce the problem in nginx-1.0.9,

[root@host-22 ~]# /usr/local/nginx/sbin/nginx -V
nginx: nginx version: nginx/1.0.9
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-51)
nginx: TLS SNI support disabled
nginx: configure arguments: --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-openssl-opt=enable-tlsext --with-http_sub_module --with-cc-opt=-O2 --with-cpu-opt=opteron --add-module=../ngx_cache_purge-1.4
[root@host-22 ~]#
[root@host-22 ~]#
[root@host-22 ~]# ps aux|grep -e CPU -e nginx
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 569 0.0 0.0 49016 6884 ? Ss 21:59 0:00 nginx: master process /usr/local/nginx/sbin/nginx
www 587 97.3 0.0 49380 7556 ? R 22:00 10:16 nginx: worker process
www 588 93.8 0.0 49380 7556 ? R 22:00 9:54 nginx: worker process
www 614 43.1 0.0 49412 7584 ? T 22:01 4:07 nginx: worker process
root 781 0.0 0.1 95984 19684 pts/0 S+ 22:06 0:00 gdb /usr/local/nginx/sbin/nginx 614
www 876 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 877 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 878 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 879 0.0 0.0 50504 8660 ? S 22:10 0:00 nginx: worker process
www 880 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 881 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 882 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
www 883 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx: worker process
root 954 0.0 0.0 61168 788 pts/1 S+ 22:10 0:00 grep -e CPU -e nginx
[root@host-22 ~]#



[root@host-22 ~]# gdb /usr/local/nginx/sbin/nginx 614
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/nginx/sbin/nginx...done.
Attaching to program: /usr/local/nginx/sbin/nginx, process 614
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized out>) at src/http/ngx_http_upstream_round_robin.c:632
632 if (reset++) {
(gdb) bt
#0 ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized out>) at src/http/ngx_http_upstream_round_robin.c:632
#1 ngx_http_upstream_get_round_robin_peer (pc=0x54ae260, data=<value optimized out>) at src/http/ngx_http_upstream_round_robin.c:425
#2 0x000000000041a99c in ngx_event_connect_peer (pc=0x54ae960) at src/event/ngx_event_connect.c:24
#3 0x000000000043d5a8 in ngx_http_upstream_connect (r=0x54c3b30, u=0x54ae250) at src/http/ngx_http_upstream.c:1103
#4 0x000000000043ee0a in ngx_http_upstream_init_request (r=0x54c3b30) at src/http/ngx_http_upstream.c:631
#5 0x00000000004354a5 in ngx_http_read_client_request_body (r=0x54c3b30, post_handler=0x43f310 <ngx_http_upstream_init>)
at src/http/ngx_http_request_body.c:154
#6 0x00000000004572d6 in ngx_http_proxy_handler (r=0x54c3b30) at src/http/modules/ngx_http_proxy_module.c:617
#7 0x000000000042b47c in ngx_http_core_content_phase (r=0x54c3b30, ph=0x583ffd8) at src/http/ngx_http_core_module.c:1365
#8 0x0000000000426967 in ngx_http_core_run_phases (r=0x54c3b30) at src/http/ngx_http_core_module.c:861
#9 0x000000000042fa66 in ngx_http_process_request (r=0x54c3b30) at src/http/ngx_http_request.c:1665
#10 0x00000000004306a4 in ngx_http_process_request_line (rev=0x5843fc0) at src/http/ngx_http_request.c:911
#11 0x0000000000419e86 in ngx_event_process_posted (cycle=<value optimized out>, posted=0x68bd88) at src/event/ngx_event_posted.c:39
#12 0x000000000041f608 in ngx_worker_process_cycle (cycle=0x560f670, data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:801
#13 0x000000000041dd69 in ngx_spawn_process (cycle=0x560f670, proc=0x41f550 <ngx_worker_process_cycle>, data=0x0, name=0x466581 "worker process",
respawn=-4) at src/os/unix/ngx_process.c:196
#14 0x000000000041ebeb in ngx_start_worker_processes (cycle=0x560f670, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:360
#15 0x000000000041ff88 in ngx_master_process_cycle (cycle=0x560f670) at src/os/unix/ngx_process_cycle.c:249
#16 0x00000000004060d9 in main (argc=1, argv=<value optimized out>) at src/core/nginx.c:405
(gdb) n
425 rrp->current = ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb) n
425 rrp->current = ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb) n
425 rrp->current = ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb)





I found i made a mistake in nginx.conf, i include a virtual host configuation like this:
upstream test_servers {
#server 10.0.7.4:80 ;
server 10.0.7.5:80 backup;
#server 10.0.7.6:80 ;
#server 10.0.7.7:80 ;
}

server {
listen 80;
server_name test.org ;

access_log /data1/logs/$host.access.log main;

location / {
include proxy.conf;
proxy_pass http://test_servers;
}


}

there was only one server in upstream,which marked 'backup'. after some test,i found this is the reason.

but when i test the nginx.conf syntax by using nginx -t,the result is ok.

[root@host-22 ~]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@host-22 ~]#

i think nginx should warn me when that situation,haha...
Subject Author Posted

nginx worker process hang,cpu load 100%

Long Wan November 13, 2011 07:06AM

Re: nginx worker process hang,cpu load 100%

Maxim Dounin November 13, 2011 07:36AM

Re: nginx worker process hang,cpu load 100%

Long Wan November 13, 2011 08:28PM

Re: nginx worker process hang,cpu load 100%

Maxim Dounin November 14, 2011 05:20AM

Re: nginx worker process hang,cpu load 100%

Long Wan November 14, 2011 09:37AM

Re: nginx worker process hang,cpu load 100%

Maxim Dounin November 14, 2011 11:00AM

Re: nginx worker process hang,cpu load 100%

Long Wan November 14, 2011 08:56PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 125
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready