Hello!

On Wed, Sep 20, 2023 at 11:55:39AM -0500, Lance Dockins wrote:

> Are there any best practices or processes for debugging sudden memory
> spikes in Nginx on production servers? We have a few very high-traffic
> servers that are encountering events where the Nginx process memory
> suddenly spikes from around 300mb to 12gb of memory before being shut down
> by an out-of-memory termination script. We don't have Nginx compiled with
> debug mode and even if we did, I'm not sure that we could enable that
> without overly taxing the server due to the constant high traffic load that
> the server is under. Since it's a server with public websites on it, I
> don't know that we could filter the debug log to a single IP either.
>
> Access, error, and info logs all seem to be pretty normal. Internal
> monitoring of the Nginx process doesn't suggest that there are major
> connection spikes either. Theoretically, it is possible that there is just
> a very large sudden burst of traffic coming in that is hitting our rate
> limits very hard and bumping the memory that Nginx is using until the OOM
> termination process closes Nginx (which would prevent Nginx from logging
> the traffic). We just don't have a good way to see where the memory in
> Nginx is being allocated when these sorts of spikes occur and are looking
> for any good insight into how to go about debugging that sort of thing on a
> production server.
>
> Any insights into how to go about troubleshooting it?

In no particular order:

- Make sure you are monitoring connection and request numbers as
reported by the stub_status module as well as memory usage.

- Check 3rd party modules you are using, if there are any - try
disabling them.

- If you are using subrequests, such as with SSI, make sure these
won't generate enormous number of subrequests.

- Check your configuration for buffer sizes and connection limits,
and make sure that your server can handle maximum memory
allocation without invoking the OOM Killer, that is:
worker_processes * worker_connections * (total amount of various
buffers as allocated per connection). If not, consider reducing
various parts of the equation.

Hope this helps.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx