Welcome! Log In Create A New Profile

Advanced

Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab
January 19, 2022 07:24AM
# Issue Summary

* After executing Nginx soft reload with "service nginx reload", nginx is able to close a lot of connections gracefully, but some connections aren't closed gracefully and Nginx is sending an RST packet. For these connections, Nginx didn't send FIN packet, and it didn't send "Connection: Close" header. The connections are HTTP/1.1 keep alive connections.

# Expected Behaviour

* After executing Nginx soft reload, Nginx is gracefully closing all connections by sending a "Connection: close" header in the response or a FIN packet.

# Supporting Data

I have tcpdump of 5 such connections where nginx didn't close the connection gracefully after nginx reload.
Here's the tcpdump - https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view?usp=sharing

If we analyse one such connection, we see that nginx didn't send FIN packet on this connection, refer this image - https://i.imgur.com/zqyLOLc.png

If we see the response of second last request, we see nginx didn't send "Connection: close" header either, refer this image - https://i.imgur.com/P2uu722.png

In this image I have plotted FIN packets sent by nginx over time - https://i.imgur.com/5lNAmnk.png

Nginx was reloaded on 2022-01-12 13:57:44 UTC.

FIN packet graph (https://i.imgur.com/5lNAmnk.png) shows that Nginx was able to close a lot of connections gracefully at the time of reload, but it wasn't able to close the 5 connections. TCP dump of which I've shared above (https://drive.google.com/file/d/1UquhmJET9i8ShEizu8453iUKpprutILV/view?usp=sharing).
Subject Author Posted

Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab January 19, 2022 07:24AM

Re: Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

Maxim Dounin January 19, 2022 08:46AM

Re: Nginx reload leading to ELB 502 on AWS Elastic Load Balancer

gab January 20, 2022 07:18AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 130
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready