Hey Maxim,

Thanks so much for the very thoughtful and detailed response! Sorry for the delay in getting back to you. I wanted to test all of this and then I got busy at work and haven't been able to get back to it.

I've run a lot of tests on this and I'm seeing some success. :)

The first test I ran was in FreeBSD just because I was curious. Lol. But I actually saw the exact same problem on it. I can send you some tests if you like but they look the same as the others do.

Now on to the partial success! But first, I'll include the results without any modifications to malloc.
--------------------------------------------------------------------------------
[root@localhost ~]# systemctl restart nginx; sleep 10; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
13.8 MiB + 750.7 MiB = 764.5 MiB nginx (3)
[root@localhost ~]# systemctl reload nginx; sleep 10; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
27.2 MiB + 1.4 GiB = 1.5 GiB nginx (3)
--------------------------------------------------------------------------------

Instead of passing the environment variables in the command line I'm going to run the tests by modifying the systemd script to use the following.
--------------------------------------------------------------------------------
Environment=MALLOC_MMAP_THRESHOLD_=16384
Environment=MALLOC_MMAP_MAX_=524288
--------------------------------------------------------------------------------

Now for the test:
--------------------------------------------------------------------------------
[root@localhost ~]# systemctl restart nginx; sleep 10; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
7.6 MiB + 905.8 MiB = 913.4 MiB nginx (3)
[root@localhost ~]# systemctl reload nginx; sleep 10; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
9.7 MiB + 905.4 MiB = 915.1 MiB nginx (3)
--------------------------------------------------------------------------------

Notice it starts up using more memory than before but it now does not double it's ram usage after a reload. I can deal with that but I'm still curious if you know why it's now using more ram. Any thoughts on that?

But when I bring this setup to one of my live servers the effect isn't quite the same. Here is a test from it with the same settings in systemd.
--------------------------------------------------------------------------------
[root@live ~]# systemctl restart nginx; sleep 60; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
21.6 MiB + 1.4 GiB = 1.5 GiB nginx (3)
[root@live ~]# systemctl reload nginx; sleep 60; ps_mem |egrep 'RAM|nginx'
Private + Shared = RAM used Program
21.9 MiB + 2.3 GiB = 2.4 GiB nginx (3)
--------------------------------------------------------------------------------

So that's definitely better but still not like the test environment. I'm sure part of the difference is that this server has real traffic and is somewhat busy. And it's using caching and probably has different buffer settings and such. So that may explain why there is an increase. But it's still very perplexing.

What are your thoughts on all of that?

And thank you again for all of the help!