On 18.08.2021 04:14, Robert Mueller wrote:
>
>> First, thanks for the patch.
>>
>> While the reuseport could cure (or hide if you will) the unbalancing you
>> see it makes sense to get better understanding what exactly is going on.
>> So far we haven't seen such weird behaviour ourself neither received
>> reports about such uneven connections distribution among nginx workers.
>>
>> Any chances you have accept_mutex and/or multi_accept? Any other ideas?
>
> Unfortunately I'm not 100% sure what's causing it, but it's pretty easy for us to reproduce even on our development machines. Just to show there's no accept_mutex or multi_accept in our config.
>
> ```
> # grep accept /etc/nginx/mail.conf
> #
> ```
>
> And here's what a cut down version of our config looks like.
>
> ```
> worker_processes auto;
> worker_shutdown_timeout 5m;
>
> events {
> use epoll;
> worker_connections 65536;
> }
> ...
> mail {
> auth_http http://unix:/var/run/nginx/mail_auth.sock:/nginx/;
> imap_client_buffer 16k;
> imap_capabilities "IMAP4" "IMAP4rev1" "LITERAL+" "ENABLE" "UIDPLUS" "SASL-IR" "NAMESPACE" "CONDSTORE" "SORT" "LIST-EXTENDED" "QRESYNC" "MOVE" "SPECIAL-USE" "CREATE-SPECIAL-USE" "IDLE";
> ssl_session_cache shared:sslcache:50m;
> ssl_session_timeout 30m;
>
> server {
> listen 10.a.b.c:993 ssl reuseport;
> auth_http_header "ServerHostname" "imap.foo";
> ssl_prefer_server_ciphers on;
> ssl_protocols ...
> ssl_ciphers ...;
> ssl_certificate ...;
> ssl_certificate_key ...;
> protocol imap;
> proxy on;
> proxy_timeout 1h;
> }
> ```
>
> With that on a development machine which has 4 vcpus we see:
>
> ```
> # ps auxw | grep nginx | grep mail
> root 3839 0.0 0.0 68472 1372 ? Ss 08:16 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 3841 0.0 0.0 95732 3572 ? S 08:16 0:01 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 3842 0.0 0.0 95732 3284 ? S 08:16 0:01 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 3843 0.0 0.0 95796 4096 ? S 08:16 0:01 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 3846 0.0 0.0 95732 3092 ? S 08:16 0:01 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> ```
>
> Now lets just create 1000 SSL connections and see how they get distributed between those procs.
>
> ```
> # perl -e 'use IO::Socket::SSL; for (1..1000) { push @s, IO::Socket::SSL->new("imap.foo:993"); } print "done\n"; sleep 1000;'
> done
> ^Z
> [3]+ Stopped
> # for i in 3841 3842 3843 3846; do echo "$i - " `ls /proc/$i/fd | wc -l`; done
> 3841 - 335
> 3842 - 295
> 3843 - 293
> 3846 - 320
> ```
>
> Reasonably even.
>
> Now lets change `listen 10.a.b.c:993 ssl reuseport` to `listen 10.a.b.c:993 ssl` and restart.
>
> ```
> # ps auxw | grep nginx | grep mail
> root 559885 0.0 0.0 68472 3104 ? Ss 21:01 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559886 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559887 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559888 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody 559889 0.0 0.3 95620 30448 ? S 21:01 0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> # perl -e 'use IO::Socket::SSL; for (1..1000) { push @s, IO::Socket::SSL->new("imap.foo:993"); } print "done\n"; sleep 1000;'
> done
> ^Z
> [5]+ Stopped
> # for i in 559886 559887 559888 559889; do echo "$i - " `ls /proc/$i/fd | wc -l`; done
> 559886 - 1054
> 559887 - 57
> 559888 - 60
> 559889 - 57
> ```
>
> And as you can see, a completely uneven distribution of connections between processes! This doesn't just occur on our development machines either (e.g. it's not related to the source IP or anything), it occurs on production systems with connections arriving from real world customers and clients scattered around the world.
>
> This is a fairly standard debian buster distribution, though we use a back ported newer kernel, and a recent version of nginx.
>
> ```
> # uname -a
> Linux xyz 5.10.0-0.bpo.4-amd64 #1 SMP Debian 5.10.19-1~bpo10+1 (2021-03-13) x86_64 GNU/Linux
> # /usr/local/nginx/sbin/nginx -v
> nginx version: nginx/1.20.1
> ```
>
> As you can see, without the reuseport option, this causes severe scalability problems for us.
>
> Even without that though, it would just be nice to have some more consistency of the `listen` options between http/stream/mail modules as well.
>
This looks weird.

We'll try to reproduce this in our lab. Thanks for the detailed script.

Maxim

--
Maxim Konovalov
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel