Welcome! Log In Create A New Profile

Advanced

Re: Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab
January 20, 2022 07:18AM
Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
>
> On Wed, Jan 19, 2022 at 07:24:20AM -0500, gab wrote:
>
> > # Issue Summary
> >
> > * After executing Nginx soft reload with "service nginx reload",
> nginx is
> > able to close a lot of connections gracefully, but some connections
> aren't
> > closed gracefully and Nginx is sending an RST packet. For these
> connections,
> > Nginx didn't send FIN packet, and it didn't send "Connection: Close"
> header.
> > The connections are HTTP/1.1 keep alive connections.
> >
> > # Expected Behaviour
> >
> > * After executing Nginx soft reload, Nginx is gracefully closing all
> > connections by sending a "Connection: close" header in the response
> or a FIN
> > packet.
> >
> > # Supporting Data
> >
> > I have tcpdump of 5 such connections where nginx didn't close the
> connection
> > gracefully after nginx reload.
> > Here's the tcpdump -
> >
> https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view
> ?usp=sharing
> >
> > If we analyse one such connection, we see that nginx didn't send FIN
> packet
> > on this connection, refer this image -
> https://i.imgur.com/zqyLOLc.png
> >
> > If we see the response of second last request, we see nginx didn't
> send
> > "Connection: close" header either, refer this image -
> > https://i.imgur.com/P2uu722.png
> >
> > In this image I have plotted FIN packets sent by nginx over time -
> > https://i.imgur.com/5lNAmnk.png
> >
> > Nginx was reloaded on 2022-01-12 13:57:44 UTC.
> >
> > FIN packet graph (https://i.imgur.com/5lNAmnk.png) shows that Nginx
> was able
> > to close a lot of connections gracefully at the time of reload, but
> it
> > wasn't able to close the 5 connections. TCP dump of which I've
> shared above
> >
> (https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/vie
> w?usp=sharing).
>
> You may want to check if the following change improves things:
>
> https://mailman.nginx.org/pipermail/nginx-devel/2022-January/014728.ht
> ml
>
> Note though that it is generally the client's responsibility (AWS
> ELB's) to retry requests in such cases. See the this thread for
> details:
>
> https://mailman.nginx.org/pipermail/nginx-devel/2021-December/014681.h
> tml
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list -- nginx@nginx.org
> To unsubscribe send an email to nginx-leave@nginx.org


Your commit will help a lot - https://hg.nginx.org/nginx/rev/96ae8e57b3dd

I see from this commit that the change will be a part of 1.21.6 release - https://hg.nginx.org/nginx/rev/57581198e51e

Thanks Maxim!
Subject Author Posted

Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab January 19, 2022 07:24AM

Re: Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

Maxim Dounin January 19, 2022 08:46AM

Re: Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab January 20, 2022 07:18AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 252
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready