Welcome! Log In Create A New Profile

Advanced

Re: Nginx prematurely closing connections when reloaded

Craig Hoover
March 25, 2024 12:34PM
What language is your upstream API written in and are you hosting in
Amazon? Even if you aren't hosting in Amazon, it seems your values are
quite low for connect, read and send.

If you are running NodeJS for the upstream, one thing I have found is that
certain values need to be offset to avoid the 502 errors:

1. nGinx:
proxy_read_timeout and proxy_send_timeout should be set higher if not equal
(60s)

2. NodeJS:
server.keepAliveTimeout = 70 * 1000;
server.headersTimeout = 75 * 1000;

keepaliveTimeout for the NodeJS app needs to be longer than the nginx
timeouts and headersTimeout needs to slightly extend keepAliveTimeout.

Not sure if this applies to other languages but that was an issue we ran
into a few years ago.

Craig

On Mon, Mar 25, 2024 at 8:32 AM Sébastien Rebecchi <srebecchi@kameleoon.com>
wrote:

> Hello
>
>
> I have an issue with nginx closing prematurely connections when reload is
> performed.
>
>
> I have some nginx servers configured to proxy_pass requests to an
> upstream group. This group itself is composed of several servers which are
> nginx themselves, and is configured to use keepalive connections.
>
> When I trigger a reload (-s reload) on an nginx of one of the servers
> which is target of the upstream, I see in error logs of all servers in
> front that connection was reset by the nginx which was reloaded.
>
>
> Here configuration of upstream group (IPs are hidden replaced by IP_X):
>
> --- BEGIN ---
>
> upstream data_api {
>
> random;
>
>
> server IP_1:80 max_fails=3 fail_timeout=30s;
>
> server IP_2:80 max_fails=3 fail_timeout=30s;
>
> server IP_3:80 max_fails=3 fail_timeout=30s;
>
> server IP_4:80 max_fails=3 fail_timeout=30s;
>
> server IP_5:80 max_fails=3 fail_timeout=30s;
>
> server IP_6:80 max_fails=3 fail_timeout=30s;
>
> server IP_7:80 max_fails=3 fail_timeout=30s;
>
> server IP_8:80 max_fails=3 fail_timeout=30s;
>
> server IP_9:80 max_fails=3 fail_timeout=30s;
>
> server IP_10:80 max_fails=3 fail_timeout=30s;
>
>
> keepalive 20;
>
> }
>
> --- END ---
>
>
> Here configuration of the location using this upstream:
>
> --- BEGIN ---
>
> location / {
>
> proxy_pass http://data_api;
>
>
> proxy_http_version 1.1;
>
> proxy_set_header Connection "";
>
>
> proxy_set_header Host $host;
>
> proxy_set_header X-Real-IP $real_ip;
>
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>
>
> proxy_connect_timeout 2s;
>
> proxy_send_timeout 6s;
>
> proxy_read_timeout 10s;
>
>
> proxy_next_upstream error timeout http_502 http_504;
>
> }
>
> --- END ---
>
>
> And here the kind of error messages I get when I reload nginx of "IP_1":
>
> --- BEGIN ---
>
> 2024/03/25 11:24:25 [error] 3758170#0: *1795895162 recv() failed (104:
> Connection reset by peer) while reading response header from upstream,
> client: CLIENT_IP_HIDDEN, server: SERVER_HIDDEN, request: "POST
> /REQUEST_LOCATION_HIDDEN HTTP/2.0", upstream: "
> http://IP_1:80/REQUEST_LOCATION_HIDDEN", host: "HOST_HIDDEN", referrer:
> "REFERRER_HIDDEN"
>
> --- END ---
>
>
> I thought -s reload was doing graceful shutdown of connections. Is it due
> to the fact that nginx can not handle that when using keepalive
> connections? Is it a bug?
>
> I am using nginx 1.24.0 everywhere, no particular
>
>
> Thank you for any help.
>
>
> Sébastien
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Nginx prematurely closing connections when reloaded

Sébastien Rebecchi March 25, 2024 08:34AM

Re: Nginx prematurely closing connections when reloaded

Craig Hoover March 25, 2024 12:34PM

Re: Nginx prematurely closing connections when reloaded

Craig Hoover March 25, 2024 12:36PM

Re: Nginx prematurely closing connections when reloaded

Igor Ippolitov March 25, 2024 01:00PM

Re: Nginx prematurely closing connections when reloaded

Sébastien Rebecchi March 26, 2024 08:42AM

Re: Nginx prematurely closing connections when reloaded

Igor Ippolitov March 28, 2024 11:42AM

Re: Nginx prematurely closing connections when reloaded

Sébastien Rebecchi March 28, 2024 02:30PM

Re: Nginx prematurely closing connections when reloaded

Igor Ippolitov March 28, 2024 07:06PM

Re: Nginx prematurely closing connections when reloaded

Sébastien Rebecchi March 29, 2024 04:42AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 181
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready