Hello!

On Thu, Aug 19, 2021 at 11:23:05AM +1000, Robert Mueller wrote:

> > Since reuseport uses hash of the source address to balance
> > incoming connections between sockets, the client can choose a
> > source port to use so the hash will direct the connection to a
> > particular socket, that is, to a particular worker process.
>
> But if a client is choosing the same source-ip:source-port, it's
> not a real client or OS TCP stack, it's some system using raw
> packets designed for attacking another system, and in that case
> there's many other attack options available. I'm not convinced
> this is a large real-world concern for most users of nginx.

It is trivial to select arbitrary source port using normal TCP
stack. Even if it wasn't, nothing can stop attackers from using
special tools.

I'm not saying this is large concern, yet this is a concern one
should be aware of.

> IMHO I would still really like to see this patch applied
> because:
>
> 1. The patch is relatively small and matches the http and stream
> modules
> 2. It makes the mail module consistent with the http and stream
> modules which both support reuseport on their listen arguments
> 3. The current situation is clearly really bad, and other users
> have reported that they're seeing the same issue. Your
> suggestion is to recompile nginx with a particular option
> disabled, but this isn't required for stream or http handlers,
> just adding reuseport is an acceptable option to fix those
> handlers, and I think it should be something mail handlers can
> do as well.

The reuseport is implemented in nginx to mitigate lock contention
issues on multiprocessor configurations in case of very high
accept rates. It is not available in mail because high accept
rates are not expected to be the case in mail protocols.

While reuseport can be misused as a balancing solution, this is not
something I can recommend, given various reuseport limitations and
issues. Notably, reuseport requires a lot of additional sockets,
which is known to be a problem in some configurations, does not
prevent use of the same address in other processes, which requires
great care when managing running server, and opens a DoS vector to
overload a particular worker process, as outlined above.

In the particular case you and others are reporting a better
solution would be to address the issue which causes bad balancing.
For now it looks like removing EPOLLEXCLUSIVE should fix things,
making misuse of reuseport unnecessary in all modules. Testing on
real servers affected by the issue is appreciated.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel