> On 19 Oct 2021, at 04:54, Maxim Dounin <mdounin@mdounin.ru> wrote:
>
> Hello!
>
> On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:
>
>>> On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin@mdounin.ru> wrote:
>>>
>>> # HG changeset patch
>>> # User Maxim Dounin <mdounin@mdounin.ru>
>>> # Date 1632717779 -10800
>>> # Mon Sep 27 07:42:59 2021 +0300
>>> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
>>> # Parent 8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
>>> SSL: SSL_sendfile() support with kernel TLS.
>>>
>>> Requires OpenSSL 3.0 compiled with "enable-ktls" option. Further, KTLS
>>> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
>>> configuration file or with "ssl_conf_command Options KTLS;" in nginx
>>> configuration.
>>>
>>> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>>> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
>>
>> I am not sure about mentioning ktls_ocf.ko in the commit message.
>> The module is only present in FreeBSD 13.0, it was removed post 13.0,
>> and the functionality is now always present in kernels with KERN_TLS:
>> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246
>> Further, it is one of many options to enable KTLS.
>> It could be better to refer to man ktls(4), instead:
>>
>> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
>>
>> (but I don't insist)
>
> I would rather keep it explicitly mentioned, since it is a
> required step on FreeBSD 13, and this is the only FreeBSD release
> with KTLS so far. I don't object adding ktls(4) reference though,
> updated with the following:
>
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> : can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"
> : to load a software backend, see man ktls(4) for details.
>

It looks good, thanks.

>>> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
>>> is recommended), and needs kernel compiled with CONFIG_TLS=y (with
>>> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
>>> the tls module needs to be loaded with "modprobe tls").
>>
>> On Linux I observe a problem sending data with short socket buffer space.
>> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
>> event methods. As per tcpdump traces, it looks like the buffer cannot
>> be pushed to the network, although it is reported as if it was sent.
>> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
>> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).
>
> You mean records? SSL buffer size limits buffering, and as a side
> effect it limits maximum size of SSL records generated, since
> nginx always uses the buffer to call SSL_write(). With
> SSL_sendfile(), it does not limit records generated by sendfile,
> since the buffer is not used for SSL_sendfile().
>
> (Just for the record, as of now there is no way to limit maximum
> record size with SSL_sendfile() (except may be by calling
> SSL_sendfile() many times with small file fragments, but this
> approach looks awful), as there are no kernel interfaces to
> control maximum record size. Further, OpenSSL disables KTLS if
> SSL_CTX_set_max_send_fragment() is used with anything other than
> 16384, see tls1_change_cipher_state(), "ktls supports only the
> maximum fragment size". I don't think this is a major problem
> though.)

Ok, it was useful to know.

>
>> It doesn't stuck starting with sndbuf 16k, so it might have something
>> with how KTLS send buffers correspond with TCP send buffers.
>> (In contrast, the FreeBSD sendfile is strictly constrained by the available
>> send buffer space and hence shouldn't have this problem.)
>> So it doesn't look like a major issue.
>
> I was able to reproduce this with sndbuf=32k over localhost on
> Ubuntu 21.04 (5.11.0-18-generic). Does not seem to happen with
> larger buffers, but might be I'm just not testing it hard enough.
>
> Over (emulated) network I was able to reproduce this with
> sndbuf=24k, but not with larger buffers.
>
> [...]
>
>> I've added additional debugging to SSL_sendfile()
>> to see that sendfile() returns EBUSY (11).
>
> Nitpicking: EAGAIN, not EBUSY. EBUSY on Linux is 16, and
> sendfile() on Linux shouldn't return EBUSY.

Yes, surely EAGAIN. Thanks for noticing.

>
> Overall, this looks like an issue in Linux KTLS implementation,
> probably related to the socket buffer size. While it would be
> good to mitigate this on our side if possible, I don't see
> anything obvious (I've tried tcp_nodelay, but it doesn't help).
>
> [...]

I think so, too.

--
Sergey Kandaurov

_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel