Welcome! Log In Create A New Profile

Advanced

Re: SO_REUSEPORT

February 01, 2019 09:22AM
>> If you've seen a
>> percentage of connections being dropped for some time - likely
>> there is another problem elsewhere.

That's definitely what I observed. It was around 50% of this customers
connections and strace on all workers (including the shutting down worker)
did not show the missed connections at the accept level (grep on unique
testing IP).

The only thing strange I was able to note was the one process remaining in
"worker is shutting down state" (it's not uncommon for us to have a few
workers hanging around for a while due to websocket or similar connections
keeping workers open). This is why I formulated this theory.

Further reloads did not resolve the issue, it took a restart of the nginx
process to get everything back to normal.

As far as I am aware no other nginx process was started on the server
(systemd manages nginx).

On Sat, Feb 2, 2019 at 1:13 AM Maxim Dounin <mdounin@mdounin.ru> wrote:

> Hello!
>
> On Fri, Feb 01, 2019 at 11:04:50AM +1100, Mathew Heard wrote:
>
> > Hit a rather strange issue today on a production service where during a
> > configuration reload (evident by the worker processes in the process of
> > being shutdown). During this reload a percentage of connections were not
> > getting accepted (and hence not processed). I was able to confirm that
> none
> > of the processes were accepting the connections.
> >
> > Our configuration includes the reuseport option so my theory was that for
> > some reason connections were still being routed to the shutting down
> > worker, which was not accepting new connections.
>
> With "listen ... reuseport" nginx creates a listening socket for
> each worker process. And on configuration reload these sockets
> are passed to the new worker processes, so there shouldn't be
> problems.
>
> The only "risky" case is reducing the number of worker process.
> Reducing the number of worker process means that some of the
> listening sockets will be closed, and on Linux this can result
> in rejecting some of the connection requests sitting in these
> sockets when these sockets are closed. (AFAIK, this is properly
> handled only on DragonFly BSD, where connection requests are
> redistributed to other sockets in such a case.)
>
> This is, however, not about "a percentage of connections", but
> about a small number of connections sitting in the listening
> socket when old worker process is instructed to exit gracefully
> and closes the listening socket.
>
> If you've changed the number of worker processes and seen several
> connections dropped - this may be the case. If you've seen a
> percentage of connections being dropped for some time - likely
> there is another problem elsewhere.
>
> In particular, one common caveat with "listen ... reuseport" is
> that listening socket no longer prevents multiple instances of
> nginx (or event different servers) from running on the same port.
> As a result, accidentally starting another nginx instance can
> easily screw up things.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

SO_REUSEPORT

splitice 349 January 31, 2019 07:06PM

Re: SO_REUSEPORT

Valentin V. Bartenev 112 February 01, 2019 09:06AM

Re: SO_REUSEPORT

splitice 129 February 01, 2019 09:08AM

Re: SO_REUSEPORT

Maxim Dounin 151 February 01, 2019 09:14AM

Re: SO_REUSEPORT

splitice 156 February 01, 2019 09:22AM

Re: SO_REUSEPORT

Maxim Dounin 121 February 01, 2019 10:50AM

Re: SO_REUSEPORT

Tomas1345 133 February 05, 2019 07:26AM

Re: SO_REUSEPORT

splitice 103 April 23, 2019 08:40PM

Re: SO_REUSEPORT

splitice 83 May 02, 2019 06:52AM

Re: SO_REUSEPORT

Maxim Dounin 73 May 03, 2019 06:54PM

Re: SO_REUSEPORT

splitice 84 May 03, 2019 07:04PM

Re: SO_REUSEPORT

Maxim Dounin 81 May 03, 2019 08:16PM

Re: SO_REUSEPORT

splitice 118 May 03, 2019 08:18PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 99
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready