> On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin@mdounin.ru> wrote:
>
> Hello!
>
> This patch series add kernel TLS / SSL_sendfile() support.
> Works on FreeBSD 13.0+ and Linux with kernel 4.13+ (at least 5.2
> is recommended, tested with 5.11).
>
> The following questions need additional testing/attention:
>
> - What about EINTR? Looks like it simply results in SSL_ERROR_WANT_WRITE,
> so might need extra checking to make sure there will be another write
> event.
>
> - What about SSL_sendfile(), early data and write blocking?
> Ref. c->ssl->write_blocked, 7431:294162223c7c by pluknet@.
> Looks like it is not a problem with SSL_sendfile(), but needs
> further checking.
>

On that particular one.

Indeed, it should not be an issue, since KTLS bypasses OpenSSL internals.
For the record, I've reproduced the original issue fixed in 294162223c7c.
For example, it could be reading discarded body sent separately in 1-RTT.
Even with the fix backed out, reading with blocked sendfile works fine.

2021/10/12 16:15:53 [debug] 38707#0: *2 SSL buf copy: 246
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL to write: 246
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_write_early_data: 1, 246
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL to sendfile: @0 1048576
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_sendfile: 45056
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL to sendfile: @45056 1003520
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_sendfile: 40960
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL to sendfile: @86016 962560
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_sendfile: 61440
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL to sendfile: @147456 901120
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_sendfile: -1
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_get_error: 3
2021/10/12 16:15:53 [debug] 38707#0: *2 http write filter 0000000802259660
2021/10/12 16:15:53 [debug] 38707#0: *2 http copy filter: -2 "/file?"
2021/10/12 16:15:53 [debug] 38707#0: *2 http finalize request: -2, "/file?" a:1,
c:2
2021/10/12 16:15:53 [debug] 38707#0: *2 event timer add: 13: 60000:707289850
2021/10/12 16:15:53 [debug] 38707#0: *2 kevent set event: 13: ft:-2 fl:0025
2021/10/12 16:15:53 [debug] 38707#0: timer delta: 1
2021/10/12 16:15:53 [debug] 38707#0: worker cycle
2021/10/12 16:15:53 [debug] 38707#0: kevent timer: 60000, changes: 1
2021/10/12 16:15:53 [debug] 38707#0: kevent events: 1
2021/10/12 16:15:53 [debug] 38707#0: kevent: 13: ft:-1 fl:0020 ff:00000000 d:138 ud:0000000802328841
2021/10/12 16:15:53 [debug] 38707#0: *2 http run request: "/file?"
2021/10/12 16:15:53 [debug] 38707#0: *2 http read discarded body
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_read_early_data: 2, 0
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_read: 10
2021/10/12 16:15:53 [debug] 38707#0: *2 SSL_read: avail:128

For comparison (and to make sure I'm testing it right),
disabling sendfile on unfixed nginx would reintroduce an error:

2021/10/12 16:33:41 [debug] 42445#0: *2 SSL_read_early_data: 2, 0
2021/10/12 16:33:41 [alert] 42445#0: *2 ignoring stale global SSL error (SSL: error:0A00010F:SSL routines::bad length) while sending response to client, client: 127.0.0.1, server: localhost, request: "GET /file HTTP/1.1", host: "localhost"
2021/10/12 16:33:41 [debug] 42445#0: *2 SSL_read: -1
2021/10/12 16:33:41 [debug] 42445#0: *2 SSL_get_error: 5

> - What about FreeBSD aio sendfile (aka SF_NODISKIO)? Might be
> easy enough to support.
>
> Review and testing appreciated.
>

--
Sergey Kandaurov

_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel