Hello!

On Tue, Sep 18, 2018 at 06:02:46AM -0400, domleb wrote:

> While running a load test that injects 10k TPS across 3 Nginx instances, we
> are seeing spikes of errors where Nginx returns HTTP 502 and logs the
> message 'no live upstreams while connecting to upstream'. There are no
> other errors logged e.g. connection errors.
>
> Also, we have a single upstream virtual IP (we use iptables to balance load
> across the backend) and according to the docs the upstream should never be
> marked as down in this case:
>
> 'If there is only a single server in a group, max_fails, fail_timeout and
> slow_start parameters are ignored, and such a server will never be
> considered unavailable'
>
> Testing locally with our config confirms this and I cannot reproduce the 'no
> live upstreams while connecting to upstream' message when simulating
> connection and read errors with a single upstream.
>
> To debug I tried enabling debug logs but under load that degraded
> performance too much. I also traced the worker process with strace and
> didn't find any socket or other other errors during the 502 spike.
>
> I was able to create this issue on Nginx 1.12.2 and 1.15.3.
>
> So given that we don't see any source error and we have a single upstream,
> I'm interested to know what other scenarios could result in a 502 with the
> log message 'no live upstreams while connecting to upstream'?

Could you please show the upstream configuration you are using?

With a single server in the upstream block, "no live upstreams"
error may happen if:

- the server is marked "down" in the configuration, or
- the server reached the max_conns limit.

Also note that "a single server" does not apply to cases when
there is a single hostname which resolves to multiple IP address
(this defines multiple servers at once).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx