Hello!

On Tue, Jan 19, 2021 at 01:33:54PM +0000, James Beal wrote:

> > Are you able to reproduce the problem without any 3rd party
> > modules? Since nginx itself does not use pipes, this looks
> > like a pagespeed problem.
>
> Not really we use about 500mb/s of bandwidth with pagespeed
> turned on and we are using about 2 gigabits a second with it
> turned off ( although I think that is more hitting the limits of
> the interconnect ).

Well, so you'll have to debug what happens then. Debugger and
strace are your friends. Most likely you'll end up fixing
pagespeed (or working on removing it from your system). Some
useful links from a quick look into pagespeed code below.

> A two minute look at the nginx source shows pipes around
> upstream I think.

These aren't OS pipes, but rather nginx code named
"ngx_event_pipe".

> Is there a method of working out where an nginx process is stuck
> ( It does not respond to the normal signals ).

As long as the process is stuck in a 3rd party module code,
spinning in a loop, the only thing you can do is to send the KILL
signal to the process, which cannot be catched or ignored and
kills the process. Or you can attach to the process with a
debugger and take a look where it spins. I suspect this will be
this code, which simply re-tries writing on EAGAIN:

https://github.com/apache/incubator-pagespeed-ngx/blob/master/src/ngx_event_connection.cc#L151

The "TODO(oschaaf): should we worry about spinning here?" line in
the code looks like exactly what you are seeing. The relevant
issue is linked in the same file:

https://github.com/apache/incubator-pagespeed-ngx/issues/1380

That is, the code is known to do not work under load for at least
three years. This basically matches my impression from previous
attempts to look into the pagespeed code: it is not expected to
work. I would really recommend to reconsider its usage. If you
need optimizations of the responses returned, consider doing this
during your deployment.

Hope this helps.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx