Hi all,

I just wanted to share the details of what I've found about this issue. Also thanks to Maxim Dounin and Reinis Rozitis who gave some really great answers!

The more I look into this the more I'm convinced this is an issue with Nginx itself. I've tested this with 3 different builds now and all have the exact same issue.

The first 2 types of servers I tested were both running Nginx 1.15.8 on Centos 7 ( with 1 of them being on 6 ). I tested about 10 of our over 100 servers. This time I tested in a default install of Debian 9 with Nginix version 1.10.3 and the issue exists there too. I just wanted to test on something completely different.

For the test, I created 50k very simple vhosts which used about 1G of RAM. Here is the ps_mem output.
94.3 MiB + 1.0 GiB = 1.1 GiB nginx (3)

After a normal reload it then uses 2x the ram:
186.3 MiB + 1.9 GiB = 2.1 GiB nginx (3)

And if I reload it again it briefly jumps up to about 4G during the reload and then goes back down to 2G.

If I instead use the "upgrade" option. In the case of Debian, service nginx upgrade, then it reloads gracefully and goes back to using 1G again.
100.8 MiB + 1.0 GiB = 1.1 GiB nginx (3)

The difference between the "reload" and "upgrade" process is basically only that reload sends a HUP signal to Nginx and upgrade sends a USR2 and then QUIT signal. What happens with all of those signals is entirely up to Nginx. It could even ignore them if chose too.

Additionally, I ran the same test with Apache. Not because I want to compare Nginx to Apache, they are different for a reason. I just wanted to test if this was a system issue. So I did the same thing on Debian 9, installed Apache and created 50k simple vhosts. It used about 800M of ram and reloading did not cause that to increase at all.

All of that leads me to these questions.

Why would anyone want to use the normal reload process to reload the Nginx configuration?
Shouldn't we always be using the upgrade process instead?
Are there any downsides to doing that?
Has anyone else noticed these issues and have you found another fix?

Look forward to hearing back and thanks in advance!