I'm running to the exact same issue, and I've done exactly the same troubleshoot, yet I don't have any more ideas of what to try ....

I'm with Ubuntu 22.04.1 LTS , Linux HOST 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Product Name: ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0 x16, Crypto and Secure Boot
[PN] Part number: MCX623106AC-CDAT

Running the latest firmware and drivers

ethtool -i enp193s0f1np1
driver: mlx5_core
version: 5.7-1.0.2
firmware-version: 22.34.4000 (MT_0000000436)
expansion-rom-version:
bus-info: 0000:c1:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

ethtool -k enp193s0f1np1 | grep tls
tls-hw-tx-offload: on
tls-hw-rx-offload: on
tls-hw-record: off [fixed]

I have almost everything the same as @liwuliu, yet I'm unable to use NIC kTLS.

I tried nginx 1.23.1, and 1.22.0
I tried openSSL 3.0.1 and 3.0.5
I tried static content only, I tried reverse proxy, yet unable to make HW kTLS work (based on TLS stats, and based on Ethtool -S stats)
SW kTLS works:

cat /proc/net/tls_stat

TlsCurrTxSw 0
TlsCurrRxSw 0
TlsCurrTxDevice 0
TlsCurrRxDevice 0
TlsTxSw 11
TlsRxSw 0
TlsTxDevice 0
TlsRxDevice 0
TlsDecryptError 0
TlsRxDeviceResync 0

inline/nic kTLS doesn't seem to work

tx_tls_encrypted_packets: 0
tx_tls_encrypted_bytes: 0
tx_tls_ooo: 0
tx_tls_dump_packets: 0
tx_tls_dump_bytes: 0
tx_tls_resync_bytes: 0
tx_tls_skip_no_sync_data: 0
tx_tls_drop_no_sync_data: 0
tx_tls_drop_bypass_req: 0
rx_tls_decrypted_packets: 0
rx_tls_decrypted_bytes: 0
rx_tls_resync_req_pkt: 0
rx_tls_resync_req_start: 0
rx_tls_resync_req_end: 0
rx_tls_resync_req_skip: 0
rx_tls_resync_res_ok: 0
rx_tls_resync_res_retry: 0
rx_tls_resync_res_skip: 0
rx_tls_err: 0
tx_tls_ctx: 0
tx_tls_del: 0
rx_tls_ctx: 0
rx_tls_del: 0
rx0_tls_decrypted_packets: 0
rx0_tls_decrypted_bytes: 0
rx0_tls_resync_req_pkt: 0
rx0_tls_resync_req_start: 0
rx0_tls_resync_req_end: 0
rx0_tls_resync_req_skip: 0
rx0_tls_resync_res_ok: 0
rx0_tls_resync_res_retry: 0
rx0_tls_resync_res_skip: 0
rx0_tls_err: 0


All the settings @liwuliu wrote, I have the same.

Only thing I'm not sure when @liwuliu wrote he made it work, if it was typo OpenSSL "3.1.0" that he said, cuz I can't find that version, so maybe he tried 3.0.1? not quite sure, Latest I was able to find is 3.0.5.

I checked his Cipher list, and its 100% exactly the same as his (/nginx/openssl-3.0.5/.openssl/bin] ./openssl ciphers)
my Nginx is built the same as well.

nginx version: nginx/1.22.0
built by gcc 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
built with OpenSSL 3.0.5 5 Jul 2022
TLS SNI support enabled
configure arguments: --with-debug --with-http_slice_module --with-http_ssl_module --with-http_realip_module --with-http_mp4_module --with-http_flv_module --with-threads --with-http_stub_status_module --with-http_secure_link_module --with-http_gzip_static_module --with-http_v2_module --with-http_gunzip_module --with-http_geoip_module --with-pcre-jit --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_random_index_module --with-http_sub_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-openssl=../openssl-3.0.5 --with-openssl-opt=enable-ktls --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC'


This is the output of strace, downloading 400MB file via TLS
strace -e trace=network -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 :
https://dpaste.com/HSU5QY2PY

This is "curl -v https://domain.com/data/1" output:
https://dpaste.com/29DSYBQU2

my nginx config is the following:

pcre_jit on;
error_log /home/logs/error.log debug;
user www-data;
worker_processes auto;
worker_rlimit_nofile 50000;
worker_cpu_affinity auto;

events {
worker_connections 50000;
multi_accept on;
}

http {
include mime.types;
# tcp_nodelay on;
# tcp_nopush on;
sendfile on;
# sendfile_max_chunk 1m;
keepalive_timeout 60;

server {

listen 443 ssl reuseport;
server_name *.domain;

ssl_conf_command Options KTLS;
ssl_certificate /usr/local/nginx/cert/certificate.cer;
ssl_certificate_key /usr/local/nginx/cert/certificate.key;
ssl_protocols TLSv1.3;

#ssl_session_cache shared:SSL:10m;
#ssl_session_timeout 5m;
#ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
#ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
#ssl_prefer_server_ciphers on;
#access_log /home/logs/access.log;
#error_log /home/logs/error.log debug;

location / {
root html;
}

}
}


I would appreciate any help.