Thanks, Sergey.

We are simulating 1000 clients. Some get cache hits, and some go upstream. So there are more than 1000 connections.

We have 24 workers running, each configured: events { worker_connections 1024; }

We are seeing the following errors from nginx:
[warn] 21151#21151: 1024 worker_connections are not enough, reusing connections
[crit] 21151#21151: accept4() failed (24: Too many open files)
[alert] 21151#21151: *15716 socket() failed (24: Too many open files) while connecting to upstream,

I am assuming the second and third error are for the OS limit. But the first seems to be from a worker process.

My assumption is that the client requests will be distributed over the 24 worker processes. So no individual worker should come anywhere close to 1000 connections.

But when I look at the process stats for the workers (ps command), I see a uneven distribution of CPU time used. Note that this is from a different run than the above logs.
UID PID PPID C STIME TTY TIME CMD
netskrt 16905 16902 2 12:19 ? 00:07:05 nginx: worker process
netskrt 16906 16902 1 12:19 ? 00:04:29 nginx: worker process
netskrt 16908 16902 1 12:19 ? 00:03:30 nginx: worker process
netskrt 16910 16902 0 12:19 ? 00:02:26 nginx: worker process
netskrt 16911 16902 0 12:19 ? 00:01:32 nginx: worker process
netskrt 16912 16902 0 12:19 ? 00:00:51 nginx: worker process
netskrt 16913 16902 0 12:19 ? 00:00:11 nginx: worker process
netskrt 16914 16902 0 12:19 ? 00:00:04 nginx: worker process
netskrt 16915 16902 0 12:19 ? 00:00:25 nginx: worker process
netskrt 16916 16902 0 12:19 ? 00:00:01 nginx: worker process
netskrt 16917 16902 0 12:19 ? 00:00:00 nginx: worker process
....

Is there anything we can configure to more evenly distribute the connections?

Thanks…

Roger


> On Jun 3, 2022, at 8:40 PM, Sergey A. Osokin <osa@freebsd.org.ru> wrote:
>
> Hi Roger,
>
> hope you're doing well.
>
> On Fri, Jun 03, 2022 at 05:38:07PM -0700, Roger Fischer wrote:
>> Hello,
>>
>> my understanding is that worker_connections applies to each worker
>> (eg. when set to 1024, 10 worker processes could handle up to 10240
>> connections).
>
> That's exactly right. Please read the following link [1] to get more
> details.
>
>> But we are seeing 1024 worker_connections are not enough, reusing
>> connections from one worker while other workers are idle.
>
> So, it's possibe to increase the number of those.
>
>> Is there something we can do to balance connections more evenly
>> across workers?
>
> Could you please add a bit more details on this. Please note, that
> there were several improvements on that topic, so please follow the
> recommendations below.
>
>> nginx version: nginx/1.19.9
>
> Recent stable version is 1.22.0, [2] so I'd recommend to update to
> that version.
>
> Thank you.
>
> References
> 1. https://nginx.org/en/docs/ngx_core_module.html#worker_connections
> 2. http://nginx.org/en/CHANGES-1.22
>
> --
> Sergey A. Osokin
> _______________________________________________
> nginx mailing list -- nginx@nginx.org
> To unsubscribe send an email to nginx-leave@nginx.org

_______________________________________________
nginx mailing list -- nginx@nginx.org
To unsubscribe send an email to nginx-leave@nginx.org