Nginx version: 1.13.6.1
1) In our use case, the Nginx is reloaded constantly. you will see lots worker process hanging at "nginx: worker process is shutting down" after couple days:
58 root 0:00 nginx: master process ./openresty/nginx/sbin/nginx -p /opt/applicatio
1029 nobody 0:22 nginx: worker process is shutting down
1030 nobody 0:27 nginx: worker process is shutting down
1041 nobody 0:54 nginx: worker process is shutting down
1054 nobody 0:37 nginx: worker process is shutting down
1131 nobody 0:02 nginx: worker process is shutting down
1132 nobody 0:02 nginx: worker process is shutting down
1215 nobody 0:50 nginx: worker process is shutting down
1216 nobody 0:53 nginx: worker process is shutting down
1515 nobody 0:23 nginx: worker process is shutting down
1516 nobody 0:47 nginx: worker process is shutting down
1533 nobody 0:03 nginx: worker process is shutting down
1534 nobody 0:16 nginx: worker process is shutting down
1598 nobody 0:00 nginx: worker process
1599 nobody 0:00 nginx: worker process
2) And if you now check some listen port on the host using netstat, you will see it is owned by the worker process:
[root@paas-controller-177-1-1-137:~]$ netstat -anp |grep 10080
tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 6288/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50827 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50833 ESTABLISHED 6300/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49411 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.40.157.154:54074 ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49715 ESTABLISHED 6299/nginx: worker
tcp6 0 0 :::10080 :::* LISTEN 6288/nginx: worker
3) So far, you would say it is not a big deal as long as the worker could serve the request correctly. but it does NOT. suppose the listen port 10080 above would proxy the request to the upstream server A initially (and for sure this port was listening by the master process), days later, it has been changed to the server B and of course the Nginx was been reloaded at the same time (and perhaps some worker was left in " is shutting down" state at that time). well now, I see this port is listening by a worker process, and it would proxy the request to the old server A.
I suspected this could be caused by the "shutting down" worker processes, so I "kill -9" all these ones and try again, but nothing changed. even reloading the Nginx did no help. I shut it down and rerun the binary, finally this worked and the request can be proxyed to the server B.
I guess maybe some cache in Nginx was doing the bad, any clue?
Thanks,
Allen