Welcome! Log In Create A New Profile

Advanced

Re: SO_REUSEPORT

April 23, 2019 08:40PM
Yesterday one of my techs reported that a production server had a nginx
worker sitting at 99.9% CPU usage in top and not accepting new connections
(but getting it's share distributed due to SO_REUSEPORT). I thought this
might be related.

The workers age was significantly older than it's peers so it appears to
have been a worker left from a previous configuration reload. It was child
to the single running master process and there was nothing of interest in
the error logs.

Just sharing this here as it sounds related.

On Tue, Feb 5, 2019 at 11:25 PM Tomas Kvasnicka <nzt4567@gmx.com> wrote:

> Hi!
>
> I just wanted to add my 2 cents here….
>
> - We are facing a very similar issue. During reloads of reuseport enabled
> configuration it sometimes happens that no workers accept new connections,
> bandwidth basically drops to 0 and it all reverts back to normal within few
> seconds. Hurts but not deadly.
> - Our nginx uses the Intel QAT card which allows only a limited number of
> user-space processes using the card at any given moment. Therefore, our HUP
> handler has been patched quite a bit to shutdown workers one-by-one and
> then use the ngx_reap_child to start a new worker with updated
> configuration.
> - We only use the reuseport option in combination with the QAT-based
> servers (servers with the patched HUP behaviour). Therefore, I am not yet
> sure if changes in the reload mechanism are the cause of the issue - still
> investigating.
> - Thanks a lot for the “ss -nltp” tip.
> - Currently testing the “iptables-drop-empty-ACKs” workaround.
> - No two nginx masters are running simultaneously.
> - Reducing the number of workers is also not happening, due to the changes
> described above - the new value of worker_processes will simply be ignored.
>
> Thanks,
> Tomas
> _______________________________________________
> nginx-devel mailing list
> nginx-devel@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

SO_REUSEPORT

splitice 400 January 31, 2019 07:06PM

Re: SO_REUSEPORT

Valentin V. Bartenev 147 February 01, 2019 09:06AM

Re: SO_REUSEPORT

splitice 169 February 01, 2019 09:08AM

Re: SO_REUSEPORT

Maxim Dounin 198 February 01, 2019 09:14AM

Re: SO_REUSEPORT

splitice 239 February 01, 2019 09:22AM

Re: SO_REUSEPORT

Maxim Dounin 157 February 01, 2019 10:50AM

Re: SO_REUSEPORT

Tomas1345 167 February 05, 2019 07:26AM

Re: SO_REUSEPORT

splitice 137 April 23, 2019 08:40PM

Re: SO_REUSEPORT

splitice 119 May 02, 2019 06:52AM

Re: SO_REUSEPORT

Maxim Dounin 102 May 03, 2019 06:54PM

Re: SO_REUSEPORT

splitice 118 May 03, 2019 07:04PM

Re: SO_REUSEPORT

Maxim Dounin 117 May 03, 2019 08:16PM

Re: SO_REUSEPORT

splitice 164 May 03, 2019 08:18PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 55
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready