Welcome! Log In Create A New Profile

Advanced

Re: Worker other than the master is listening to the socket

April 19, 2019 03:16AM
Nginx version: 1.13.6.1

1) In our use case, the Nginx is reloaded constantly. you will see lots worker process hanging at "nginx: worker process is shutting down" after couple days:

58 root 0:00 nginx: master process ./openresty/nginx/sbin/nginx -p /opt/applicatio
1029 nobody 0:22 nginx: worker process is shutting down
1030 nobody 0:27 nginx: worker process is shutting down
1041 nobody 0:54 nginx: worker process is shutting down
1054 nobody 0:37 nginx: worker process is shutting down
1131 nobody 0:02 nginx: worker process is shutting down
1132 nobody 0:02 nginx: worker process is shutting down
1215 nobody 0:50 nginx: worker process is shutting down
1216 nobody 0:53 nginx: worker process is shutting down
1515 nobody 0:23 nginx: worker process is shutting down
1516 nobody 0:47 nginx: worker process is shutting down
1533 nobody 0:03 nginx: worker process is shutting down
1534 nobody 0:16 nginx: worker process is shutting down
1598 nobody 0:00 nginx: worker process
1599 nobody 0:00 nginx: worker process


2) And if you now check some listen port on the host using netstat, you will see it is owned by the worker process:

[root@paas-controller-177-1-1-137:~]$ netstat -anp |grep 10080
tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 6288/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50827 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50833 ESTABLISHED 6300/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49411 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.40.157.154:54074 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49715 ESTABLISHED 6299/nginx: worker
tcp6 0 0 :::10080 :::* LISTEN 6288/nginx: worker


3) So far, you would say it is not a big deal as long as the worker could serve the request correctly. but it does NOT. suppose the listen port 10080 above would proxy the request to the upstream server A initially (and for sure this port was listening by the master process), days later, it has been changed to the server B and of course the Nginx was been reloaded at the same time (and perhaps some worker was left in " is shutting down" state at that time). well now, I see this port is listening by a worker process, and it would proxy the request to the old server A.
I suspected this could be caused by the "shutting down" worker processes, so I "kill -9" all these ones and try again, but nothing changed. even reloading the Nginx did no help. I shut it down and rerun the binary, finally this worked and the request can be proxyed to the server B.
I guess maybe some cache in Nginx was doing the bad, any clue?

Thanks,
Allen
Subject Author Posted

Worker other than the master is listening to the socket

allenhe April 10, 2019 02:11AM

Re: Worker other than the master is listening to the socket

allenhe April 19, 2019 03:16AM

Re: Worker other than the master is listening to the socket

Maxim Dounin April 22, 2019 10:32AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 179
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready