Welcome! Log In Create A New Profile

Advanced

Debugging Nginx Memory Spikes on Production Servers

Lance Dockins
September 20, 2023 12:56PM
Are there any best practices or processes for debugging sudden memory
spikes in Nginx on production servers? We have a few very high-traffic
servers that are encountering events where the Nginx process memory
suddenly spikes from around 300mb to 12gb of memory before being shut down
by an out-of-memory termination script. We don't have Nginx compiled with
debug mode and even if we did, I'm not sure that we could enable that
without overly taxing the server due to the constant high traffic load that
the server is under. Since it's a server with public websites on it, I
don't know that we could filter the debug log to a single IP either.

Access, error, and info logs all seem to be pretty normal. Internal
monitoring of the Nginx process doesn't suggest that there are major
connection spikes either. Theoretically, it is possible that there is just
a very large sudden burst of traffic coming in that is hitting our rate
limits very hard and bumping the memory that Nginx is using until the OOM
termination process closes Nginx (which would prevent Nginx from logging
the traffic). We just don't have a good way to see where the memory in
Nginx is being allocated when these sorts of spikes occur and are looking
for any good insight into how to go about debugging that sort of thing on a
production server.

Any insights into how to go about troubleshooting it?

--
Lance Dockins
_______________________________________________
nginx mailing list
nginx@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 20, 2023 12:56PM

Re: Debugging Nginx Memory Spikes on Production Servers

Maxim Dounin September 20, 2023 03:08PM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 20, 2023 11:38PM

Re: Debugging Nginx Memory Spikes on Production Servers

Dmitry Volyntsev September 21, 2023 02:46AM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 21, 2023 09:52AM

Re: Debugging Nginx Memory Spikes on Production Servers

Dmitry Volyntsev September 21, 2023 06:02PM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 21, 2023 07:42PM

Re: Debugging Nginx Memory Spikes on Production Servers

Dmitry Volyntsev September 21, 2023 08:48PM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 22, 2023 09:36AM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 26, 2023 11:32AM

Re: Debugging Nginx Memory Spikes on Production Servers

Dmitry Volyntsev September 26, 2023 06:52PM

Re: Debugging Nginx Memory Spikes on Production Servers

Lance Dockins September 26, 2023 09:22PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 310
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready