Welcome! Log In Create A New Profile

Advanced

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Sergey Kandaurov
October 19, 2021 06:50AM
> On 19 Oct 2021, at 04:54, Maxim Dounin <mdounin@mdounin.ru> wrote:
>
> Hello!
>
> On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:
>
>>> On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin@mdounin.ru> wrote:
>>>
>>> # HG changeset patch
>>> # User Maxim Dounin <mdounin@mdounin.ru>
>>> # Date 1632717779 -10800
>>> # Mon Sep 27 07:42:59 2021 +0300
>>> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
>>> # Parent 8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
>>> SSL: SSL_sendfile() support with kernel TLS.
>>>
>>> Requires OpenSSL 3.0 compiled with "enable-ktls" option. Further, KTLS
>>> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
>>> configuration file or with "ssl_conf_command Options KTLS;" in nginx
>>> configuration.
>>>
>>> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>>> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
>>
>> I am not sure about mentioning ktls_ocf.ko in the commit message.
>> The module is only present in FreeBSD 13.0, it was removed post 13.0,
>> and the functionality is now always present in kernels with KERN_TLS:
>> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246
>> Further, it is one of many options to enable KTLS.
>> It could be better to refer to man ktls(4), instead:
>>
>> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
>>
>> (but I don't insist)
>
> I would rather keep it explicitly mentioned, since it is a
> required step on FreeBSD 13, and this is the only FreeBSD release
> with KTLS so far. I don't object adding ktls(4) reference though,
> updated with the following:
>
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> : can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"
> : to load a software backend, see man ktls(4) for details.
>

It looks good, thanks.

>>> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
>>> is recommended), and needs kernel compiled with CONFIG_TLS=y (with
>>> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
>>> the tls module needs to be loaded with "modprobe tls").
>>
>> On Linux I observe a problem sending data with short socket buffer space.
>> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
>> event methods. As per tcpdump traces, it looks like the buffer cannot
>> be pushed to the network, although it is reported as if it was sent.
>> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
>> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).
>
> You mean records? SSL buffer size limits buffering, and as a side
> effect it limits maximum size of SSL records generated, since
> nginx always uses the buffer to call SSL_write(). With
> SSL_sendfile(), it does not limit records generated by sendfile,
> since the buffer is not used for SSL_sendfile().
>
> (Just for the record, as of now there is no way to limit maximum
> record size with SSL_sendfile() (except may be by calling
> SSL_sendfile() many times with small file fragments, but this
> approach looks awful), as there are no kernel interfaces to
> control maximum record size. Further, OpenSSL disables KTLS if
> SSL_CTX_set_max_send_fragment() is used with anything other than
> 16384, see tls1_change_cipher_state(), "ktls supports only the
> maximum fragment size". I don't think this is a major problem
> though.)

Ok, it was useful to know.

>
>> It doesn't stuck starting with sndbuf 16k, so it might have something
>> with how KTLS send buffers correspond with TCP send buffers.
>> (In contrast, the FreeBSD sendfile is strictly constrained by the available
>> send buffer space and hence shouldn't have this problem.)
>> So it doesn't look like a major issue.
>
> I was able to reproduce this with sndbuf=32k over localhost on
> Ubuntu 21.04 (5.11.0-18-generic). Does not seem to happen with
> larger buffers, but might be I'm just not testing it hard enough.
>
> Over (emulated) network I was able to reproduce this with
> sndbuf=24k, but not with larger buffers.
>
> [...]
>
>> I've added additional debugging to SSL_sendfile()
>> to see that sendfile() returns EBUSY (11).
>
> Nitpicking: EAGAIN, not EBUSY. EBUSY on Linux is 16, and
> sendfile() on Linux shouldn't return EBUSY.

Yes, surely EAGAIN. Thanks for noticing.

>
> Overall, this looks like an issue in Linux KTLS implementation,
> probably related to the socket buffer size. While it would be
> good to mitigate this on our side if possible, I don't see
> anything obvious (I've tried tcp_nodelay, but it doesn't help).
>
> [...]

I think so, too.

--
Sergey Kandaurov

_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

[PATCH 0 of 2] KTLS / SSL_sendfile() support

Maxim Dounin 996 September 27, 2021 10:06AM

[PATCH 1 of 2] Style: added missing "static" specifiers

Maxim Dounin 145 September 27, 2021 10:06AM

[PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Maxim Dounin 197 September 27, 2021 10:06AM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Sergey Kandaurov 245 October 18, 2021 11:28AM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Sergey Kandaurov 133 October 18, 2021 06:10PM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Maxim Dounin 130 October 18, 2021 09:56PM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Sergey Kandaurov 160 October 19, 2021 06:50AM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Sergey Kandaurov 53 November 24, 2022 04:22AM

Re: [PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

Maxim Dounin 85 November 24, 2022 01:32PM

Re: [PATCH 0 of 2] KTLS / SSL_sendfile() support

Sergey Kandaurov 132 October 12, 2021 12:50PM

Re: [PATCH 0 of 2] KTLS / SSL_sendfile() support

Maxim Dounin 197 October 13, 2021 09:28AM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 87
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready