March 07, 2019 01:33PM
Hi all,

I just wanted to share the details of what I've found about this issue. Also thanks to Maxim Dounin and Reinis Rozitis who gave some really great answers!

The more I look into this the more I'm convinced this is an issue with Nginx itself. I've tested this with 3 different builds now and all have the exact same issue.

The first 2 types of servers I tested were both running Nginx 1.15.8 on Centos 7 ( with 1 of them being on 6 ). I tested about 10 of our over 100 servers. This time I tested in a default install of Debian 9 with Nginix version 1.10.3 and the issue exists there too. I just wanted to test on something completely different.

For the test, I created 50k very simple vhosts which used about 1G of RAM. Here is the ps_mem output.
94.3 MiB + 1.0 GiB = 1.1 GiB nginx (3)

After a normal reload it then uses 2x the ram:
186.3 MiB + 1.9 GiB = 2.1 GiB nginx (3)

And if I reload it again it briefly jumps up to about 4G during the reload and then goes back down to 2G.

If I instead use the "upgrade" option. In the case of Debian, service nginx upgrade, then it reloads gracefully and goes back to using 1G again.
100.8 MiB + 1.0 GiB = 1.1 GiB nginx (3)

The difference between the "reload" and "upgrade" process is basically only that reload sends a HUP signal to Nginx and upgrade sends a USR2 and then QUIT signal. What happens with all of those signals is entirely up to Nginx. It could even ignore them if chose too.

Additionally, I ran the same test with Apache. Not because I want to compare Nginx to Apache, they are different for a reason. I just wanted to test if this was a system issue. So I did the same thing on Debian 9, installed Apache and created 50k simple vhosts. It used about 800M of ram and reloading did not cause that to increase at all.

All of that leads me to these questions.

Why would anyone want to use the normal reload process to reload the Nginx configuration?
Shouldn't we always be using the upgrade process instead?
Are there any downsides to doing that?
Has anyone else noticed these issues and have you found another fix?

Look forward to hearing back and thanks in advance!
Subject Author Posted

Possible memory leak?

wkbrad February 28, 2019 01:43PM

Re: Possible memory leak?

Maxim Dounin February 28, 2019 03:08PM

Re: Possible memory leak?

wkbrad February 28, 2019 03:54PM

Re: Possible memory leak?

Maxim Dounin February 28, 2019 04:50PM

Re: Possible memory leak?

wkbrad February 28, 2019 05:06PM

RE: Possible memory leak?

Reinis Rozitis February 28, 2019 05:28PM

Re: Possible memory leak?

Alceu R. de Freitas Jr. via nginx February 28, 2019 06:02PM

Re: RE: Possible memory leak?

wkbrad February 28, 2019 06:44PM

Re: Possible memory leak?

wkbrad March 07, 2019 01:33PM

Re: Possible memory leak?

Anoop Alias March 07, 2019 08:06PM

Re: Possible memory leak?

wkbrad March 07, 2019 09:35PM

Re: Possible memory leak?

Anoop Alias March 07, 2019 10:10PM

Re: Possible memory leak?

wkbrad March 08, 2019 10:39AM

Re: Possible memory leak?

Anoop Alias March 08, 2019 06:54PM

Re: Possible memory leak?

gariac March 08, 2019 01:10AM

Re: Possible memory leak?

wkbrad March 08, 2019 10:42AM

Re: Possible memory leak?

gariac March 08, 2019 08:08PM

Re: Possible memory leak?

wkbrad March 11, 2019 04:37PM

Re: Possible memory leak?

Anoop Alias March 12, 2019 09:56AM

Re: Possible memory leak?

Maxim Dounin March 12, 2019 10:40AM

Re: Possible memory leak?

Anoop Alias March 12, 2019 11:00AM

Re: Possible memory leak?

Maxim Dounin March 12, 2019 11:48AM

Re: Possible memory leak?

wkbrad March 12, 2019 02:09PM

Re: Possible memory leak?

Maxim Dounin March 12, 2019 09:58PM

Re: Possible memory leak?

Peter Booth via nginx March 13, 2019 01:44AM

Re: Possible memory leak?

Anoop Alias March 13, 2019 01:54AM

Re: Possible memory leak?

wkbrad March 20, 2019 06:41PM

Re: Possible memory leak?

Maxim Dounin March 21, 2019 09:46AM

Re: Possible memory leak?

Maxim Dounin March 21, 2019 09:48AM

Re: Possible memory leak?

wkbrad March 21, 2019 11:29AM

Re: Possible memory leak?

Maxim Dounin March 21, 2019 02:18PM

Re: Possible memory leak?

wkbrad March 21, 2019 05:55PM

Re: Possible memory leak?

Maxim Dounin March 25, 2019 08:40AM

Re: Possible memory leak?

wkbrad March 27, 2019 02:54PM

Re: Possible memory leak?

Maxim Dounin March 27, 2019 04:26PM

Re: Possible memory leak?

gariac March 13, 2019 01:58AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 93
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready