Hello!

On Sun, Oct 01, 2023 at 12:39:20AM +0530, Vijay Kumar Kamannavar wrote:

> Hello.
>
> I am using nginx reverse proxy for s3 presigned urls.
> I am running nginx as a container using nginx:1.25.2 debian image. My host
> has 16 Core and 32GB.
>
> Below is the nginx configuration.
>
> user nginx;
> worker_processes 14;
> pid /run/nginx.pid;
> worker_rlimit_nofile 40000;
> events {
> worker_connections 1024;
> }
> http {
> upstream s3_backend {
> server <mybucket>.s3.amazonaws.com:443;
> keepalive 10;
> }
>
> log_format combined_ssl '$remote_addr - $remote_user [$time_local] '
> '"$request" $status $body_bytes_sent '
> '"$http_referer" "$http_user_agent" '
> '$ssl_protocol/$ssl_cipher '
> '$ssl_session_reused';
> proxy_ssl_session_reuse on;
> proxy_ssl_server_name on;

Just a side note: with "proxy_ssl_server_name" nginx uses the name
as written in the "proxy_pass" directive during SSL handshake. In
your case, it is "s3_backend". Most likely, this is not what you
want to happen. If you do not want to send the name (which is the
default), consider removing the "proxy_ssl_server_name" directive
from your configuration. If you want to redefine the name,
consider using the "proxy_ssl_name" directive
(http://nginx.org/r/proxy_ssl_name).

[...]

> But in the log /var/log/nginx/ssl_debug.log, I see SSL Handshake every time
> when I request an S3 object via proxy using S3presigned URLs.

SSL handshake is expected to happen on each SSL connection
establishment. Depending on whether there is a cached SSL
session, SSL handshake can be full or abbreviated - with
abbreviated being more efficient.

From the logs you've provided it looks like SSL handshakes to
upstream servers is your concern.

If you want to avoid SSL handshakes to upstream servers on
proxying, focus on keeping connections to upstream servers alive:
this should be possible as long as the upstream server supports it
and it is configured on nginx side (and it looks to be already
configured in your config).

Alternatively, check that SSL sessions are being reused - this
normally happens automatically for statically defined upstream
servers (unless explicitly disabled with "proxy_ssl_session_reuse
off;", see http://nginx.org/r/proxy_ssl_session_reuse).

>
> Below is the log I see every time for every request.
>
> 2023/09/30 18:07:19 [debug] 36#36: *9 event timer add: 22: 60000:721858477
> 2023/09/30 18:07:19 [debug] 36#36: *9 http finalize request: -4,
> "/blob/zte3odk1ymnl@CIBC-2mb
> /singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930>
> 2023/09/30 18:07:19 [debug] 36#36: *9 http request count:2 blk:0
> 2023/09/30 18:07:19 [debug] 36#36: *9 http run request:
> "/blob/zte3odk1ymnl@CIBC-2mb
> /singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930%2Fus-eas>
> 2023/09/30 18:07:19 [debug] 36#36: *9 http upstream check client, write
> event:1, "/blob/zte3odk1ymnl@CIBC-2mb/singleurl0"
> 2023/09/30 18:07:19 [debug] 36#36: *9 http upstream request:
> "/blob/zte3odk1ymnl@CIBC-2mb
> /singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930%2Fu>
> 2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request handler
> 2023/09/30 18:07:19 [debug] 36#36: *9 malloc: 000055ED330A1DD0:96
> 2023/09/30 18:07:19 [debug] 36#36: *9 upstream SSL server name: "s3_backend"
> 2023/09/30 18:07:19 [debug] 36#36: *9 set session: 0000000000000000

Note: here nginx restores previously saved SSL session, yet there
are none. This suggests this is a first request to the upstream
server in question within the given nginx worker process.

> 2023/09/30 18:07:19 [debug] 36#36: *9 tcp_nodelay
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
> 2023/09/30 18:07:19 [debug] 36#36: *9 save session: 000055ED330FBAC0

Note: here nginx saves the SSL session which was established
during the handshake. This SSL session is expected to be used
during following handshakes in the same worker process.

> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: 1
> 2023/09/30 18:07:19 [debug] 36#36: *9 SSL: TLSv1.2, cipher:
> "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128)
> Mac=AEAD"

Note: here nginx logs handshake details. This handshake does not
reuse an SSL session, since there were none. If there was an SSL
session and it was correctly reused during the SSL handshake, the
next log line would be:

2023/09/30 18:07:19 [debug] 36#36: *9 SSL reused session

Check the following SSL handshakes in the same worker process to
see if sessions are actually reused or not.

Most likely, these sessions are properly reused, and everything
already works as it should.

> 2023/09/30 18:07:19 [debug] 36#36: *9 *http upstream ssl handshake*:
> "/blob/zte3odk1ymnl@CIBC-2mb
> /singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F202309>
> 2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request
> 2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request body
>
> If I run 4K clients using a simulator,I will see 100% CPU in the nginx
> container.I believe if we cache SSL sessions then SSL handshake for every
> request will be avoided hence we may not have high CPU at nginx container.
>
> Can you please help how to achieve SSL Cache? how to make sure the CPU is
> not high? Is there any reason why the CPU is high other than SSL Handshake.

As outlined above, most likely SSL session reuse to upstream
servers is already working properly in your setup.

Note though that SSL is generally costly, and you are using it for
both client connections and upstream connections. Depending on
the certificates being used, ciphers being used and so on costs
might vary, and there might be a room for improvement.

Hope this helps.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx