Hello,

We have setup NGINX as a network proxy and performs 2-way TLS on either side, upstream and downstream. We noticed that randomly the container is giving 502s when connecting to upstream. The error logged is - "upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream"

Given that most of the times SSL handshake is correct I do not want to assume that its a configuration problem. Attaching it below for reference:
Also we see multiple attempts made to different upstream IPs registered against an upstream host. This can be inferred from

$upstream_status = "-, 502 : 502 : 200"
$upstream_addr = ", IP1:8443 : IP2:8443 : IP1:8443"
As we can see, at times its the same upstream IP that is tried twice within the same request and it succeeds

The NGINX server block taking in traffic is default. In other words all domains end up in this server block. I have some other logic of treating them differently, but yes this would imply a lot of connections being made into this server section.
I am not sure if NGINX has any limits to the upstream connections that can be made from a single server block.

What could be the reason for these sporadic SSL errors? "20:unable to get local issuer certificate" implies that NGINX could not find its local truststore to verify the incoming server certificate against. But its right there...

server {
listen 9080 default_server proxy_protocol;
listen 9081 default_server ssl proxy_protocol;

server_name "_";

proxy_ssl_certificate "/ssl/client.crt";
proxy_ssl_certificate_key "/ssl/client.key";
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate "/pki/ca/cacerts.pem";
proxy_ssl_verify on;
proxy_ssl_verify_depth 3;
proxy_ssl_protocols "TLSv1.2";

----
}