Hello!
On Wed, Aug 18, 2021 at 11:14:25AM +1000, Robert Mueller wrote:
> > First, thanks for the patch.
> >
> > While the reuseport could cure (or hide if you will) the unbalancing you
> > see it makes sense to get better understanding what exactly is going on.
> > So far we haven't seen such weird behaviour ourself neither received
> > reports about such uneven connections distribution among nginx workers.
> >
> > Any chances you have accept_mutex and/or multi_accept? Any other ideas?
>
> Unfortunately I'm not 100% sure what's causing it, but it's
> pretty easy for us to reproduce even on our development
> machines. Just to show there's no accept_mutex or multi_accept
> in our config.
[...]
> Now lets change `listen 10.a.b.c:993 ssl reuseport` to `listen 10.a.b.c:993 ssl` and restart.
>
> ```
> # ps auxw | grep nginx | grep mail
> root 559885 0.0 0.0 68472 3104 ? Ss 21:01 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559886 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559887 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559888 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559889 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> # perl -e 'use IO::Socket::SSL; for (1..1000) { push @s, IO::Socket::SSL->new("imap.foo:993"); } print "done\n"; sleep 1000;'
> done
> ^Z
> [5]+ Stopped
> # for i in 559886 559887 559888 559889; do echo "$i - " `ls /proc/$i/fd | wc -l`; done
> 559886 - 1054
> 559887 - 57
> 559888 - 60
> 559889 - 57
> ```
>
> And as you can see, a completely uneven distribution of
> connections between processes! This doesn't just occur on our
> development machines either (e.g. it's not related to the source
> IP or anything), it occurs on production systems with
> connections arriving from real world customers and clients
> scattered around the world.
Could you please test if compiling with --with-cc-opt="-DNGX_HAVE_EPOLLEXCLUSIVE=0"
improves things, notably on production systems? In my limited
testing it seems to be improve things, and if this is indeed the
case, we can consider removing use of EPOLLEXCLUSIVE.
Note well that the above test may, and probably will, result in
uneven distribution in many cases. Things are, however, expected
to improve over time, since new connections are more likely to be
accepted by idle worker processes and not by worker processes
which are already close to 100% CPU usage. That is, unless worker
processes are restarted regularly in your setup for some reason,
you probably should observe balancing improvements over time. The
behaviour might depend on EPOLLEXCLUSIVE though, and it would be
interesting to observe what happens in practice.
[...]
> As you can see, without the reuseport option, this causes severe
> scalability problems for us.
I tend to think that reuseport is a bad option for load balancing
between worker processes, as it can be easily tricked by an outside
actor to select a particular worker process, and this opens an
obvious DoS attack vector.
--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
