I have an nginx server that is behaving oddly under slight DDoS attack. The memory usage grows exponentially in just a few secs until max system ram is exhausted and OOM killer kicks in. But when the DDoS stops the memory footprint remains at that max level - I have to manually stop/start nginx to recover.

I've been able to replicate this using h2load (i.e. alot of traffic with multiplexing) and I have found that a too high setting of keepalive_requests is the culprit. Lowering that number prevents this runaway memory exhaustion.

So... that means the memory exhaustion I initially saw could be due to the keepalive connection cache. I'd like to prove this theory out by running the DDoS, letting nginx max out system memory, then analyze a stack trace or memory dump to see if is indeed the keepalive connection cache using all the memory but I am unfamiliar with C debugging tools and how to do this, any hints?