Hello!
On Fri, May 27, 2016 at 03:15:25AM -0400, onecrazymonkey wrote:
> It has been a difficult topic to research. The nginx instance is doing
> nothing more than what the subject stated. It reverse proxies to a backend,
> load balanced set of web app instances and terminates SSL for a large number
> of unique domains each with their own SSL cert. Here's a `ps aux` of nginx
> running after a clean start and zero (out of rotation) traffic.
>
> root 20 0 676052 598224 1848 S 0.0 16.5 0:00.06 nginx
>
> nginx 20 0 675552 597204 1228 S 0.0 16.5 0:00.44 nginx
>
> nginx 20 0 675552 596612 636 S 0.0 16.5 0:00.36 nginx
>
> Looking at that process list, nginx is using about 676mb of RAM for ~400
> vhosts each with their own unique SSL cert for a unique domain. Here's an
> example of a vhost server config. They're all generated based on the same
> base template:
>
> server {
> listen 443 ssl proxy_protocol;
> server_name <uniquehostname> www.<uniquehostname>;
> access_log /var/log/nginx/access_vhost_443.log accesslog;
> error_log /var/log/nginx/error_vhost_443.log warn;
> real_ip_header proxy_protocol;
>
> ssl on;
> ssl_certificate /etc/nginx/ssl/<uniquehostname>/<uniquehostname>.crt;
> ssl_certificate_key /etc/nginx/ssl/<uniquehostname>/<uniquehostname>.key;
>
> ssl_stapling on;
> ssl_stapling_verify on;
>
> resolver internal-dns.vpc valid=60s;
Note: by using using own resolver in each server{} you add more
flexibility, but waste memory and other resources. Consider using
single resolver defined at http{} level instead.
> set $internal "upstream-load-balancer.vpc";
> location / {
> if ($denied) {
> return 444;
> }
> proxy_pass http://$internal;
> }
> }
>
> Now, this wouldn't be all that bad. 1.69mb of memory per vhost isn't
> horrible, high, but not unsustainable. However, if I do `nginx -s reload` or
> restart via systemd service...
>
> root 20 0 1370188 1.176g 3240 S 0.0 33.4 0:14.98 nginx
>
> nginx 20 0 1370192 1.175g 2584 S 0.3 33.4 2:39.95 nginx
>
> nginx 20 0 1370192 1.175g 2584 S 1.7 33.4 2:28.42 nginx
>
> It doubles the memory consumption! It never goes up or down drastically
> again. It's as if it duplicates and never frees or releases unless you do a
> restart.
As nginx allocates a new configuration before releasing the old
one, and this doubles memory consumption of the master process for a
while. The old configuration is then freed, but in many cases
this memory isn't really returned to kernel by the allocator, and
hence seen in various stats.
> This was tested on a handful of AWS EC2 instance types using vanilla Centos7
> and both nginx 1.6.3 (stable in centos repos) and nginx 1.10.0 (nginx.org
> repo).
>
> In summary, my questions are thus:
> - Is it normal for nginx to use ~1.7mb per SSL vhost?
Not really. This is likely due to a particular configuration you
are using: own resolver in each vhost, own logs, etc. Consider
moving common settings to the http{} level, it should somewhat reduce
memory consumption.
Just for the reference: a simple test with 10k SSL vhosts like:
server { listen 8443; ssl on; server_name 1.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }
server { listen 8443; ssl on; server_name 2.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }
....
server { listen 8443; ssl on; server_name 10000.foo; ssl_certificate test.crt; ssl_certificate_key test.key; }
takes about 200mb of memory (on a 32-bit host). That is, about
20k of memory per vhost.
With more options it will take more memory, but 1.7mb is a bit too
many - you are doing something wrong.
Just a guess: if you use ssl_trusted_certificate at http{} level,
make sure you are loading only needed certificates, not a full
OS-provided bundle.
> - Is there a way to reduce that memory usage?
Yes, see various hints above. In summary: configure things at
http{} level where possible, avoid memory-hungry things you don't
need.
> - Am I the only one that experiences the doubling of nginx memory usage
> after a nginx reload?
> - Is that a bug?
No and no, see above.
--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
