Hello!

On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:

> > On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin@mdounin.ru> wrote:
> >
> > # HG changeset patch
> > # User Maxim Dounin <mdounin@mdounin.ru>
> > # Date 1632717779 -10800
> > # Mon Sep 27 07:42:59 2021 +0300
> > # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
> > # Parent 8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
> > SSL: SSL_sendfile() support with kernel TLS.
> >
> > Requires OpenSSL 3.0 compiled with "enable-ktls" option. Further, KTLS
> > needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
> > configuration file or with "ssl_conf_command Options KTLS;" in nginx
> > configuration.
> >
> > On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> > can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
>
> I am not sure about mentioning ktls_ocf.ko in the commit message.
> The module is only present in FreeBSD 13.0, it was removed post 13.0,
> and the functionality is now always present in kernels with KERN_TLS:
> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246
> Further, it is one of many options to enable KTLS.
> It could be better to refer to man ktls(4), instead:
>
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
>
> (but I don't insist)

I would rather keep it explicitly mentioned, since it is a
required step on FreeBSD 13, and this is the only FreeBSD release
with KTLS so far. I don't object adding ktls(4) reference though,
updated with the following:

: On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
: can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"
: to load a software backend, see man ktls(4) for details.

> > On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
> > is recommended), and needs kernel compiled with CONFIG_TLS=y (with
> > CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
> > the tls module needs to be loaded with "modprobe tls").
>
> On Linux I observe a problem sending data with short socket buffer space.
> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
> event methods. As per tcpdump traces, it looks like the buffer cannot
> be pushed to the network, although it is reported as if it was sent.
> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).

You mean records? SSL buffer size limits buffering, and as a side
effect it limits maximum size of SSL records generated, since
nginx always uses the buffer to call SSL_write(). With
SSL_sendfile(), it does not limit records generated by sendfile,
since the buffer is not used for SSL_sendfile().

(Just for the record, as of now there is no way to limit maximum
record size with SSL_sendfile() (except may be by calling
SSL_sendfile() many times with small file fragments, but this
approach looks awful), as there are no kernel interfaces to
control maximum record size. Further, OpenSSL disables KTLS if
SSL_CTX_set_max_send_fragment() is used with anything other than
16384, see tls1_change_cipher_state(), "ktls supports only the
maximum fragment size". I don't think this is a major problem
though.)

> It doesn't stuck starting with sndbuf 16k, so it might have something
> with how KTLS send buffers correspond with TCP send buffers.
> (In contrast, the FreeBSD sendfile is strictly constrained by the available
> send buffer space and hence shouldn't have this problem.)
> So it doesn't look like a major issue.

I was able to reproduce this with sndbuf=32k over localhost on
Ubuntu 21.04 (5.11.0-18-generic). Does not seem to happen with
larger buffers, but might be I'm just not testing it hard enough.

Over (emulated) network I was able to reproduce this with
sndbuf=24k, but not with larger buffers.

[...]

> I've added additional debugging to SSL_sendfile()
> to see that sendfile() returns EBUSY (11).

Nitpicking: EAGAIN, not EBUSY. EBUSY on Linux is 16, and
sendfile() on Linux shouldn't return EBUSY.

Overall, this looks like an issue in Linux KTLS implementation,
probably related to the socket buffer size. While it would be
good to mitigate this on our side if possible, I don't see
anything obvious (I've tried tcp_nodelay, but it doesn't help).

[...]

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel