I figured it out and thought I'd post back for anyone else looking at this post in the future.

My problem had nothing to do with the PKCS#11 engine. It persisted when I pointed proxy_ssl_certificate_key directly at the non-encrypted, password-less rsa key file.

Instead, the problem was SNI. By default, Nginx uses the inbound request's Host header as the upstream SNI name. Since I was hitting Nginx with curl on localhost, it ended up sending "localhost" as the upstream virtual host. It's even in the debug error log:

2020/02/05 07:40:28 [error] 25199#25199: *1 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: ::1, server: _, request: "GET /upstream HTTP/1.1", upstream: "https://10.16.1.21:443/", host: "localhost"

Since the upstream server does not have localhost as its SNI name, the TLS connection failed to get established. By fixing the value for SNI it went through:

proxy_ssl_server_name on;
proxy_ssl_name upstream.example.org:443;

I had to do a similar thing for the upstream HTTP Host header, which was also being set to the value of the incoming request (again, localhost for me):

proxy_set_header Host upstream.example.org:443;

Now to get the full PKCS#11 uri for the Yubikey I ran:

$ p11tool --provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so --list-privkeys --login
Token 'PIV Card Holder pin (PIV_II)' with URL 'pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=00000000;token=PIV%20Card%20Holder%20pin%20%28PIV_II%29' requires user PIN
Enter PIN:
Object 0:
URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=00000000;token=PIV%20Card%20Holder%20pin%20%28PIV_II%29;id=%01;object=PIV%20AUTH%20key;type=private
Type: Private key
Label: PIV AUTH key
Flags: CKA_WRAP/UNWRAP; CKA_PRIVATE; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
ID: 01

Prepending that with "engine:pkcs11:" and plugging that into proxy_ssl_certificate_key:

proxy_ssl_certificate /etc/nginx/ssl/cert.pem;
proxy_ssl_certificate_key "engine:pkcs11:pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=00000000;token=PIV%20Card%20Holder%20pin%20%28PIV_II%29;id=%01;object=PIV%20AUTH%20key;type=private;pin-value=123456";

And that made the whole thing work. Note that the client certificate itself is still read from a file as proxy_ssl_certificate does not support pkcs11 uri's.

I can now access the remote TLS server through the local proxy:

$ curl http://localhost/foo/bar


Erik van Zijst