According to the documentation (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_certificate_key), proxy_ssl_certificate_key supports syntax for ssl-engine specific backends:

> The value engine:name:id can be specified instead of the file (1.7.9), which loads a secret key with a specified id from
> the OpenSSL engine name.

which implies that at least for the private key we should be able to configure a pluggable ssl engine backend.

I've got my private key loaded in aYubikey and have the pkcs11 engine loaded in openssl:

$ openssl engine -t pkcs11
(pkcs11) pkcs11 engine
[ available ]

However, when I specify:

location /upstream {
proxy_pass https://10.16.1.21:443/;
proxy_ssl_certificate /etc/nginx/ssl/cert.pem;
proxy_ssl_certificate_key "engine:pkcs11:pkcs11:id=%01;type=private";
}

and hit the endpoint with debug error logging turned on, it fails during the upstream TLS handshake:

2020/02/05 07:40:28 [debug] 25199#25199: *1 http upstream request: "/upstream?"
2020/02/05 07:40:28 [debug] 25199#25199: *1 http upstream send request handler
2020/02/05 07:40:28 [debug] 25199#25199: *1 malloc: 000055AB2AB745C0:72
2020/02/05 07:40:28 [debug] 25199#25199: *1 set session: 0000000000000000
2020/02/05 07:40:28 [debug] 25199#25199: *1 tcp_nodelay
2020/02/05 07:40:28 [debug] 25199#25199: *1 SSL_do_handshake: -1
2020/02/05 07:40:28 [debug] 25199#25199: *1 SSL_get_error: 2
2020/02/05 07:40:28 [debug] 25199#25199: *1 SSL handshake handler: 0
2020/02/05 07:40:28 [debug] 25199#25199: *1 SSL_do_handshake: -1
2020/02/05 07:40:28 [debug] 25199#25199: *1 SSL_get_error: 5
2020/02/05 07:40:28 [error] 25199#25199: *1 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: ::1, server: _, request: "GET /upstream HTTP/1.1", upstream: "https://10.16.1.21:443/", host: "localhost"


Cheers,
Erik van Zijst