Hello!

On Fri, Apr 30, 2021 at 11:14:48AM +0200, Charlie Kilo wrote:

> correction.. if i count with
> ss -l -t |grep http | wc -l
> we have around 58340 listening sockets.. at least on that machine..

Since "listen ... reuseport;" implies a separate listening socket
for each worker process, that's expected with 50 worker processes
and 1200 "listen ... reuseport" in the configuration.

> On Fri, Apr 30, 2021 at 8:27 AM Charlie Kilo <krikkiteer@gmail.com> wrote:
>
> > Thanks a lot for the hints so far.. to provide the further info and answer
> > the questions..
> >
> > getconf ARG_MAX shows 2097152
> > ulimit -s shows 8192
> > setting it to unlimited, doesn't change anything (also not with prlimit)
> > wc -c /proc/<pid>/environ shows 1949
> >
> > it seems on a regular machine we have around 1200 listening sockets and do indeed use "reuseport"

Thanks, so all seem to be using the defaults.

Looking more closely at Linux limits, it seems that you are
hitting the MAX_ARG_STRLEN limit. It limits single argument
length (and a single environment variable length), and hardcoded
to be 32 pages, that is, almost always 128k.

With 128k limit nginx should be able to pass about 20k listening
sockets, so with 58340 sockets you are well above the limit.

An obvious workaround would be to reduce the amount of listening
sockets - either by dropping the "reuseport", or by reducing the
the number of listening sockets (normally just one on the wildcard
address is enough), or by reducing the number of worker processes.

It would be interesting to know why you are using such a large
number of listening sockets with reuseport enabled. If the use
case is reasonable, we probably should consider implementing some
workaround for the 128k limit.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx