Welcome! Log In Create A New Profile

Advanced

Re: SO_REUSEPORT

May 02, 2019 06:52AM
Got a little bit further and confirmed this is definitely to do with the
binary upgrade.

www-data 988 99.9 0.7 365124 122784 ? R Jan30 131740:46 nginx:
worker process
root 2800 0.0 1.0 361828 165044 ? Ss Jan05 27:54 nginx:
master process /usr/sbin/nginx -g daemon on; master_process on;

2800 is nginx.old, also (nginx/1.15.8) as we did 2 builds with slightly
different compile options.

The processes do not respond to nice kill signals, only a -9 was able to
kill it.

On Wed, Apr 24, 2019 at 10:38 AM Mathew Heard <mat999@gmail.com> wrote:

> Yesterday one of my techs reported that a production server had a nginx
> worker sitting at 99.9% CPU usage in top and not accepting new connections
> (but getting it's share distributed due to SO_REUSEPORT). I thought this
> might be related.
>
> The workers age was significantly older than it's peers so it appears to
> have been a worker left from a previous configuration reload. It was child
> to the single running master process and there was nothing of interest in
> the error logs.
>
> Just sharing this here as it sounds related.
>
> On Tue, Feb 5, 2019 at 11:25 PM Tomas Kvasnicka <nzt4567@gmx.com> wrote:
>
>> Hi!
>>
>> I just wanted to add my 2 cents here….
>>
>> - We are facing a very similar issue. During reloads of reuseport enabled
>> configuration it sometimes happens that no workers accept new connections,
>> bandwidth basically drops to 0 and it all reverts back to normal within few
>> seconds. Hurts but not deadly.
>> - Our nginx uses the Intel QAT card which allows only a limited number of
>> user-space processes using the card at any given moment. Therefore, our HUP
>> handler has been patched quite a bit to shutdown workers one-by-one and
>> then use the ngx_reap_child to start a new worker with updated
>> configuration.
>> - We only use the reuseport option in combination with the QAT-based
>> servers (servers with the patched HUP behaviour). Therefore, I am not yet
>> sure if changes in the reload mechanism are the cause of the issue - still
>> investigating.
>> - Thanks a lot for the “ss -nltp” tip.
>> - Currently testing the “iptables-drop-empty-ACKs” workaround.
>> - No two nginx masters are running simultaneously.
>> - Reducing the number of workers is also not happening, due to the
>> changes described above - the new value of worker_processes will simply be
>> ignored.
>>
>> Thanks,
>> Tomas
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
>
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

SO_REUSEPORT

splitice 346 January 31, 2019 07:06PM

Re: SO_REUSEPORT

Valentin V. Bartenev 112 February 01, 2019 09:06AM

Re: SO_REUSEPORT

splitice 128 February 01, 2019 09:08AM

Re: SO_REUSEPORT

Maxim Dounin 151 February 01, 2019 09:14AM

Re: SO_REUSEPORT

splitice 154 February 01, 2019 09:22AM

Re: SO_REUSEPORT

Maxim Dounin 121 February 01, 2019 10:50AM

Re: SO_REUSEPORT

Tomas1345 132 February 05, 2019 07:26AM

Re: SO_REUSEPORT

splitice 102 April 23, 2019 08:40PM

Re: SO_REUSEPORT

splitice 82 May 02, 2019 06:52AM

Re: SO_REUSEPORT

Maxim Dounin 73 May 03, 2019 06:54PM

Re: SO_REUSEPORT

splitice 83 May 03, 2019 07:04PM

Re: SO_REUSEPORT

Maxim Dounin 81 May 03, 2019 08:16PM

Re: SO_REUSEPORT

splitice 117 May 03, 2019 08:18PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 86
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready