Hi,

Several times recently, we have seen our production nginx memory usage flare up a hundred-fold, from its normal ~42 MB to 3-4 GB, for 20 minutes to an hour or so, and then recover. There is not a spike in number of connections, just memory use, so whatever causes this, it does not seem to be an increase in concurrency.

The obvious thing to suspect for this is our app's newest change, which involves streaming responses proxied from an upstream (via uwsgi_pass); these responses can get very large and run for many minutes, pumping hundreds of megabytes each. But I expect nginx memory use for these requests to be bounded by uwsgi_buffers (shouldn't it be?) -- and indeed I cannot reproduce the problem by making such requests, even copying the exact ones that are being made when the memory spike occurs. In my tests, the responses get buffered as they should be, and delivered normally, without memory growth.

So, what is a good way to investigate what causes all this memory to be suddenly allocated? Is there a way of introspecting/logging nginx memory allocation edge cases like this? (Is there documentation on this which I didn't find?)

During the memory spike (and before and after), connections are around 270, of which around 20-25 are writing and the rest are waiting.

Our nginx config has:

worker_processes 2;
events {
worker_connections 1024;
}
http {
uwsgi_buffers 64 8k;
proxy_buffers 64 8k;
}

All other *_buffers settings are at their defaults, on a 64-bit machine.

Accounting for the above buffers, other buffer defaults, and some uwsgi_cache and proxy_cache zones, and estimating on the high side (e.g. all buffer budgets fully used for each active connection) for 270 connections, here is my rough sketch of the memory uses I am aware of, probably with some misunderstandings along the way:

22 * keys_zone=10m for uwsgi_cache_path + proxy_cache_path = 220 MB
uwsgi_buffers: 64 * 8kB * 270 = 135 MB
proxy_buffers: 64 * 8kB * 270 = 135 MB
gzip_buffers: 16 * 8kB * 270 = 34 MB
output_buffers: 1 * 32kB * 270 = 8.6 MB
large_client_header_buffers: 4 * 8kB * 270 = 8.6 MB
ssl_buffer_size: 1 * 16k * 270 = 4.3 MB
client_body_buffer_size: 1 * 16kB * 270 = 4.3 MB
client_header_buffer_size: 1 * 1kB * 270 = 0.3 MB
access_log buffers: 24 * 8kB + 1 * 32k = 0.2 MB

This comes out to 546 MB tops ... what big memory use cases are missing in this picture?

(And are these buffer size upper-bound estimates significantly incorrect?)

This is in Nginx 1.6.0 on Linux 3.2.57 64-bit ... specifically:

$ nginx -V
nginx version: nginx/1.6.0
built by gcc 4.7.2 (Debian 4.7.2-5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' --with-ipv6

$ uname -a
Linux ren2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux

Thanks, best regards,

Gulli