Welcome! Log In Create A New Profile

Advanced

nginx KTLS and HTTP/2 performance degradation

Lyuben Stoev
December 02, 2021 07:06AM
Hello,
    I have tested the nginx with the patch
https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() support
with kernel TLS.) following the nginx blog article
https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/
And it sort of works, but I have bad performance when making HTTP/2
requests. If I made a HTTP/1.1 request there is 30-35% increase in
performance as the Nginx blog article stated, but when I changed the
request to use HTTP/2 the request was 40% slower than an ordinary nginx
without KTLS enabled. Does anyone have such perfomance degradation with
nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and
kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic).
The nginx vritual host is the same as in the Nginx blog article with
exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4
are used.
The KTLS seems to work, because the strace and debug logs show it. Just
the sstrange thing is when using HTTP2, the sendfile syscalls look:
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1418218] => [1426410], 8192) = 8192
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1426410] => [1434602], 8192) = 8192
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1434602] => [1442794], 8192) = 8192
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1442794] => [1450986], 8192) = 8192
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1450986] => [1459178], 8192) = 8192
    write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
    sendfile(39, 131, [1459178] => [1467370], 8192) = 8192

It is always 8K and there are thousands of sendfile syscalls....

And when the HTTP/1.1 is used the sendfile syscall changes the window:

    sendfile(32, 33, [586170368] => [586694656], 487571456) = 524288
    sendfile(32, 33, [586694656], 487047168) = -1 EAGAIN (Resource
temporarily unavailable)
    epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}],
512, 59711) = 1
    sendfile(32, 33, [586694656] => [594673664], 487047168) = 7979008
    sendfile(32, 33, [594673664] => [595001344], 479068160) = 327680
    sendfile(32, 33, [595001344], 478740480) = -1 EAGAIN (Resource
temporarily unavailable)
    epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}],
512, 60000) = 1
    sendfile(32, 33, [595001344] => [604028928], 478740480) = 9027584
    sendfile(32, 33, [604028928] => [604356608], 469712896) = 327680
    sendfile(32, 33, [604356608] => [604749824], 469385216) = 393216
    sendfile(32, 33, [604749824] => [605192192], 468992000) = 442368

Does anyone receive similar results?

root@srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
100 1024M  100 1024M    0     0  1170M      0 --:--:-- --:--:-- --:--:--
1168M
root@srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
100 1024M  100 1024M    0     0  1197M      0 --:--:-- --:--:-- --:--:--
1196M
root@srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
100 1024M  100 1024M    0     0   340M      0  0:00:03  0:00:03
--:--:--  340M
root@srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
100 1024M  100 1024M    0     0   338M      0  0:00:03  0:00:03
--:--:--  338M

srv-prod ~ # h2load -n 1 -c 1 -t 1 https://mytests.local/1G
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 100% done
finished in 7.64s, 0.13 req/s, 134.11MB/s
requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.00GB (1074922031) total, 492B (492) headers (space savings
21.66%), 1.00GB (1073741824) data
                     min         max         mean sd        +/- sd
time for request:      7.63s       7.63s       7.63s         0us 100.00%
time for connect:    13.46ms     13.46ms     13.46ms         0us 100.00%
time to 1st byte:    15.05ms     15.05ms     15.05ms         0us 100.00%
req/s           :       0.13        0.13        0.13        0.00 100.00%

srv-prod ~ # h2load --h1 -n 1 -c 1 -t 1 https://mytests.local/1G
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: http/1.1
progress: 100% done
finished in 2.65s, 0.38 req/s, 386.41MB/s
requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.00GB (1073742546) total, 631B (631) headers (space savings
0.00%), 1.00GB (1073741824) data
                     min         max         mean sd        +/- sd
time for request:      2.64s       2.64s       2.64s         0us 100.00%
time for connect:    13.72ms     13.72ms     13.72ms         0us 100.00%
time to 1st byte:    14.86ms     14.86ms     14.86ms         0us 100.00%
req/s           :       0.38        0.38        0.38        0.00 100.00%


With regards,
Lyuben Stoev

_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

nginx KTLS and HTTP/2 performance degradation

Lyuben Stoev 421 December 02, 2021 07:06AM

Re: nginx KTLS and HTTP/2 performance degradation

Maxim Dounin 138 December 02, 2021 08:08AM

Re: nginx KTLS and HTTP/2 performance degradation

Vadim Fedorenko 137 December 02, 2021 07:34PM

Re: nginx KTLS and HTTP/2 performance degradation

Maxim Dounin 123 December 02, 2021 11:12PM

RE: nginx KTLS and HTTP/2 performance degradation

NCviQwilt 45 September 08, 2022 10:00AM

Re: nginx KTLS and HTTP/2 performance degradation

Maxim Dounin 63 September 08, 2022 11:14PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 73
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready