Добрый день! Такая непонятная штука стала происходить (время появления не зафиксировали)
Есть nginx в котором:
worker_processes 28;
worker_rlimit_nofile 20000;
В случае выполнения nignx reload старые ооочень долго не завершаются. Настройка worker_shutdown_timeout 1m выставлена, но особо не помогает.
Через какое-то время появляются ошибки Too many open files. Если посмотреть по открытым файлам для каждого процесса то картина следующая:
несколько
nginx: worker process
Limit Soft Limit Hard Limit Units
Max open files 20000 20000 files
pid 43587
Currently open files: 12510
и очень много:
nginx: worker process is shutting down
Limit Soft Limit Hard Limit Units
Max open files 15240 15240 files
pid 35319
Currently open files: 217
nginx: worker process is shutting down
Limit Soft Limit Hard Limit Units
Max open files 15240 15240 files
pid 35311
Currently open files: 223
Если посмотреть список процессов, то видно что нормальных воркеров мало, у каждого перекос по файлам и такие ошибки.
nobody 1752 18.0 5.2 2752480 1712492 ? S 18:40 4:13 nginx: worker process is shutting down
nobody 6034 36.3 5.2 2752480 1740768 ? S 17:38 30:45 nginx: worker process is shutting down
nobody 6035 8.1 4.2 2752480 1399368 ? S 17:38 6:54 nginx: worker process is shutting down
nobody 6040 5.4 4.1 2752480 1359280 ? S 17:38 4:35 nginx: worker process is shutting down
nobody 6049 12.5 4.6 2752480 1539268 ? S 17:38 10:35 nginx: worker process is shutting down
nobody 6050 31.3 5.2 2752480 1739804 ? S 17:38 26:31 nginx: worker process is shutting down
nobody 6052 3.3 4.0 2752480 1343572 ? S 17:38 2:51 nginx: worker process is shutting down
nobody 6054 5.3 4.1 2752480 1360352 ? S 17:38 4:33 nginx: worker process is shutting down
nobody 6055 26.2 5.4 2752480 1790544 ? S 17:38 22:13 nginx: worker process is shutting down
nobody 6058 0.0 3.7 2752480 1227076 ? S 17:38 0:03 nginx: cache manager process
root 6168 0.0 3.9 2752480 1293176 ? Ss 13:24 0:16 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
nobody 10656 48.4 5.2 2752480 1719536 ? S 18:45 8:36 nginx: worker process is shutting down
nobody 10657 31.1 4.4 2752480 1460292 ? S 18:45 5:31 nginx: worker process is shutting down
nobody 10658 70.8 5.2 2752480 1739444 ? R 18:45 12:33 nginx: worker process
nobody 10659 70.7 5.2 2752480 1739476 ? R 18:45 12:32 nginx: worker process
nobody 10660 10.7 4.1 2752480 1377328 ? S 18:45 1:54 nginx: worker process is shutting down
nobody 11053 25.3 4.3 2752480 1440108 ? S 18:45 4:26 nginx: worker process is shutting down
nobody 11054 51.3 5.3 2752480 1756036 ? S 18:45 8:59 nginx: worker process is shutting down
nobody 11055 2.2 4.0 2752480 1324696 ? S 18:45 0:23 nginx: worker process is shutting down
nobody 11056 38.6 4.8 2752480 1604476 ? S 18:45 6:45 nginx: worker process is shutting down
nobody 11057 17.9 4.2 2752480 1400344 ? S 18:45 3:07 nginx: worker process is shutting down
nobody 16049 9.6 4.1 2752480 1373540 ? S 18:48 1:26 nginx: worker process is shutting down
nobody 19782 29.2 5.4 2752480 1784320 ? S 18:18 13:08 nginx: worker process is shutting down
nobody 29881 7.8 4.7 2752480 1571664 ? S 18:24 3:06 nginx: worker process is shutting down
nobody 32145 22.2 5.3 2752480 1757248 ? S 17:53 15:40 nginx: worker process is shutting down
nobody 33168 19.8 5.3 2752480 1748384 ? S 18:25 7:33 nginx: worker process is shutting down
nobody 36049 94.5 5.1 2752480 1702212 ? R 18:59 3:29 nginx: worker process
nobody 40384 14.9 4.1 2489628 1365948 ? S 16:22 24:07 nginx: worker process is shutting down
root 43195 0.0 0.0 13956 2188 pts/0 S+ 19:03 0:00 grep --color=auto nginx
nobody 45130 22.6 5.2 2752480 1715320 ? S 18:32 6:58 nginx: worker process is shutting down
nobody 48277 3.8 4.1 2489628 1369324 ? S 16:58 4:48 nginx: worker process is shutting down
nobody 49924 6.1 4.2 2489628 1411852 ? S 16:59 7:37 nginx: worker process is shutting down
nobody 49931 2.7 4.1 2489628 1368808 ? S 16:59 3:22 nginx: worker process is shutting down
nobody 49943 8.3 4.4 2489628 1453684 ? S 16:59 10:18 nginx: worker process is shutting down
nobody 49944 9.1 4.3 2489628 1444924 ? S 16:59 11:18 nginx: worker process is shutting down
nobody 49945 3.8 4.3 2489628 1436344 ? S 16:59 4:46 nginx: worker process is shutting down
nobody 50490 8.6 5.3 2752480 1768892 ? S 18:35 2:25 nginx: worker process is shutting down
nobody 55442 24.4 5.3 2752480 1753424 ? S 18:38 6:02 nginx: worker process is shutting down
Версия nignx 1.12.1 собрана так:
nginx version: nginx/1.12.1
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.2j 26 Sep 2016 (running with OpenSSL 1.0.2i 22 Sep 2016)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --http-client-body-temp-path=/run/shm/body_temp --http-proxy-temp-path=/run/shm/proxy_temp --http-fastcgi-temp-path=/run/shm/fastcgi_temp --http-uwsgi-temp-path=/run/shm/uwsgi_temp --http-scgi-temp-path=/run/shm/scgi_temp --with-http_ssl_module --with-http_realip_module --with-http_sub_module --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_stub_status_module --with-mail --with-http_v2_module --with-http_geoip_module --with-select_module --with-http_auth_request_module --with-poll_module --with-debug
Встречалось ли такое?
