Welcome! Log In Create A New Profile

Advanced

Re: Inquiry Regarding Handling of QUIC Connections During Nginx Reload

Roman Arutyunyan
February 26, 2024 06:50AM
Hi,

On Sun, Feb 25, 2024 at 03:53:23PM +0800, 上勾拳 wrote:
> Hello,
>
> I hope this email finds you well. I am writing to inquire about the status
> of an issue I have encountered regarding the handling of QUIC connections
> when Nginx is reloaded.
>
> Recently, I delved into the Nginx eBPF implementation, specifically
> examining how QUIC connection packets are handled, especially during Nginx
> reloads. My focus was on ensuring that existing QUIC connection packets are
> correctly routed to the appropriate worker after a reload, and the Nginx
> eBPF prog have done this job perfectly.
>
> However, I observed that during a reload, new QUIC connections might be
> directed to the old worker. The underlying problem stems from the fact that
> new QUIC connections fail to match the eBPF reuseport socket map. The
> kernel default logic then routes UDP packets based on the hash UDP 4-tuple
> in the reuseport group socket array. Since the old worker's listen FDs
> persist in the reuseport group socket array (reuse->socks), there is a
> possibility that the old worker may still be tasked with handling new QUIC
> connections.
>
> Given that the old worker should not process new requests, it results in
> the old worker not responding to the QUIC new connection. Consequently,
> clients have to endure the handshake timeout and retry the connection,
> potentially encountering the old worker again, leading to an ineffective
> cycle.
>
> In my investigation, I came across a thread in the nginx-devel mailing list
> [https://www.mail-archive.com/nginx-devel@nginx.org/msg10627.html], where
> it was mentioned that there would be some work addressing this issue.
>
> Considering my limited experience with eBPF, I propose a potential
> solution. The eBPF program could maintain another eBPF map containing only
> the listen sockets of the new worker. When the eBPF program calls
> `bpf_sk_select_reuseport` and receives `-ENOENT`, it could utilize this new
> eBPF map with the hash UDP 4-tuple to route the new QUIC connection to the
> new worker. This approach aims to circumvent the kernel logic routing the
> packet to the shutting down worker since removing the old worker's listen
> socket from the reuseport group socket array seems unfeasible. Not sure
> about whether this solution is a good idea and I also wonder if there are
> other solutions for this. I would appreciate any insights or updates you
> could provide on this matter. Thank you for your time and consideration.

It is true QUIC in nginx does not handle reloads well. This is a known issue
and we are working on improving it. I made an effort a while back to address
QUIC reloads in nginx:

https://mailman.nginx.org/pipermail/nginx-devel/2023-January/thread.html#16073

Here patch #3 implements ePBF-based solution and patch #4 implements
client sockets-based solution. The client sockets require extra worker process
privileges to bind to listen port, which is a serious problem for a typical
nginx configuration. The ePBF solution does not seem to have any problems,
but we still need more feedback to proceed with this. If you apply all 4
patches, make sure you disable client sockets using
--without-quic_client_sockets. Otherwise just apply the first 3 patches.

Here's a relevant trac ticket:

https://trac.nginx.org/nginx/ticket/2528

--
Roman Arutyunyan
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

Inquiry Regarding Handling of QUIC Connections During Nginx Reload

上勾拳 227 February 25, 2024 02:54AM

Re: Inquiry Regarding Handling of QUIC Connections During Nginx Reload

Roman Arutyunyan 97 February 26, 2024 06:50AM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 178
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready