Welcome! Log In Create A New Profile

Advanced

limit_conn is dropping valid connections and causing memory leaks on nginx reload

September 30, 2017 06:05AM
Hello,

We are running the nginx 1.13.5 with HTTP/2 in a proxy_pass proxy_cache configuration with clients having relatively long open connections. Our system does automatic reloads for any new configuration and we recently introduced a limit_conn to some of the config files. After that, I've started noticing a rapid drop in connections and outgoing network every-time the system would perform a configuration reload. Even stranger, on every reload the memory usage would go up for about 1-2GB until ultimately everything crashed if the reloads were too frequent. The memory usage did go down after old workers were released, but that could take up to 30 minutes, while the configuration could get reloaded up to twice per minute.

We used the following configuration as recommended by pretty much any example:
limit_conn_zone $binary_remote_addr zone=1234con:10m;
limit_conn zone1234con 10;

I was able to verify the connection drop by doing a simple ab test, for example, I would run ab -c 100 -n -k 1000 https://127.0.0.1/file.bin
990 of the connections went through, however, 10 would still be active. Immediately after the reload, those would get dropped as well. Adding -r option would help the problem, but that doesn't fix our problem.

Finally, after I tried to create a workaround, I've configured the limit zone to:
limit_conn_zone "v$binary_remote_addr" zone=1234con:10m;

Suddenly everything magically started to work. The connections were not being dropped, the limit worked as expected and even more surprisingly the memory usage was not going up anymore. I've been tearing my hair out almost all day yesterday trying to figure this out. While I was very happy to see this resolved, I am now confused as to why nginx behaves in such a way.

I'm thinking this might likely be a bug, so I'm just wondering if anyone could explain why it is happening or has a similar problem.

Thank you!
Subject Author Posted

limit_conn is dropping valid connections and causing memory leaks on nginx reload

Dejan Grofelnik Pelzel September 30, 2017 06:05AM

Re: limit_conn is dropping valid connections and causing memory leaks on nginx reload

Anoop Alias September 30, 2017 06:14AM

Re: limit_conn is dropping valid connections and causing memory leaks on nginx reload

Lucas Rolff September 30, 2017 06:16AM

Re: limit_conn is dropping valid connections and causing memory leaks on nginx reload

Dejan Grofelnik Pelzel September 30, 2017 08:47AM

Re: limit_conn is dropping valid connections and causing memory leaks on nginx reload

Francis Daly October 01, 2017 07:40PM

Re: limit_conn is dropping valid connections and causing memory leaks on nginx reload

Valentin V. Bartenev October 02, 2017 08:54AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 83
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready