Hello,

I need your help, I have a problem with my Nginx server : all the Nginx processes are consuming a huge amount of memory and it sets off the OOM Killer.
I don't understand why Nginx uses this much memory, it seems odd, I expected the CPU to be the first problem instead of memory.

My Nginx server is just a reverse proxy, to all kind of backend. SSL can be offloaded or not, it depends on the backend.

I have a huge amount of vhosts with SSL on this Nginx (~15K) and the server have a lot of trafic (for me at least !) .
The /stub_status tell me the following thing at this moment :

Active connections: 3867
server accepts handled requests
2350039 2350039 5489648
Reading: 0 Writing: 332 Waiting: 3605

Netdata (https://github.com/netdata/netdata) tell me that it handle between 300 and 600 req/s.

[root@nginxsrv nginx]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

[root@nginxsrv nginx]# nginx -V
nginx version: nginx/1.17.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.1.0k 28 May 2019
TLS SNI support enabled
configure arguments: --add-module=/usr/local/src/ngx_brotli --add-module=/usr/local/src/kyprizel-testcookie-nginx-module
--with-threads --with-openssl=/usr/local/src/openssl --with-http_v2_module --with-http_flv_module --with-ipv6 --with-http_mp4_module
--sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --with-http_realip_module --with-http_ssl_module --http-client-body-temp-path=/eacc/nginx_client
--http-proxy-temp-path=/eacc/nginx_proxy --http-fastcgi-temp-path=/eacc/nginx_fastcgi --with-http_stub_status_module

The server have a lot of ram, 120G and the CPU is an Intel Xeon E5-2697 CPU. The CPU is not used that much.

A typical vhost file is quite simple, it's just a proxy_pass to a backend, with some hack to do a retry is we first get and error message (based on proxy_intercept_errors). http2 is on. I don't use any caches.

Things I have tested or I suspect to be an issue :
- I reduced the worker_processes to 6. I had issue with the frequent reloading of Nginx with an upper value. I suspect that it's more difficult for Nginx to reload properly if a lot of processes are spawned (correct me if i'm wrong)
- I have gzip on and gzip_proxied to any. I suspect the compression to be an issue, I reduced the buffers value to gzip_buffers 32 4k; and gzip_comp_level to 1
- I reduced the ssl_session_cache from 10m to 1m
- I reduced the keep_alive timeout too (30)
- Reloading Nginx often seems to be a problem too, the memory usage seems higher during the reloading task. The reloading is quite slow too, probably because of the number of vhosts file to load (if someone know how to speed up this, it interest me !)
- I have a lot of limit_conn configuration
- ssl_stapling is off, I had issue with it with the DNS resolution.

Do you have idea on what can cause a high memory usage on Nginx ?

Thanks,
Alexis