I've managed to recompile nginx with enabled debug. Here is the part of successfull conection by Client #1 track:
2013/02/05 14:08:23 [debug] 38701#0: *119 accept: <MY IP ADDRESS> fd:3
2013/02/05 14:08:23 [debug] 38701#0: *119 event timer add: 3: 60000:2856497512
2013/02/05 14:08:23 [debug] 38701#0: *119 kevent set event: 3: ft:-1 fl:0025
2013/02/05 14:08:23 [debug] 38701#0: *119 malloc: 28805200:660
2013/02/05 14:08:23 [debug] 38701#0: *119 malloc: 28834400:1024
2013/02/05 14:08:23 [debug] 38701#0: *119 posix_memalign: 28860000:4096 @16
2013/02/05 14:08:23 [debug] 38701#0: *119 http check ssl handshake
2013/02/05 14:08:23 [debug] 38701#0: *119 https ssl handshake: 0x16
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL server name: "test.local"
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_do_handshake: -1
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_get_error: 2
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL handshake handler: 0
2013/02/05 14:08:23 [debug] 38701#0: *119 verify:1, error:0, depth:1, subject:"/C=AU /ST=Some-State/O=Internet Widgits Pty Ltd/CN=rootca",issuer: "/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=rootca"
2013/02/05 14:08:23 [debug] 38701#0: *119 verify:1, error:0, depth:0, subject:"/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=Client #1",issuer: "/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=rootca"
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_do_handshake: 1
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL: TLSv1, cipher: "AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1"
2013/02/05 14:08:23 [debug] 38701#0: *119 http process request line
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_read: -1
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_get_error: 2
2013/02/05 14:08:23 [debug] 38701#0: *119 http process request line
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_read: 1
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_read: 524
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_read: -1
2013/02/05 14:08:23 [debug] 38701#0: *119 SSL_get_error: 2
2013/02/05 14:08:23 [debug] 38701#0: *119 http request line: "GET / HTTP/1.1"
And here is the part of unsuccessfull conection by Client #2 track:
2013/02/05 13:51:34 [debug] 38701#0: *112 accept: <MY_IP_ADDRESS> fd:3
2013/02/05 13:51:34 [debug] 38701#0: *112 event timer add: 3: 60000:2855488975
2013/02/05 13:51:34 [debug] 38701#0: *112 kevent set event: 3: ft:-1 fl:0025
2013/02/05 13:51:34 [debug] 38701#0: *112 malloc: 28805200:660
2013/02/05 13:51:34 [debug] 38701#0: *112 malloc: 28834400:1024
2013/02/05 13:51:34 [debug] 38701#0: *112 posix_memalign: 28860000:4096 @16
2013/02/05 13:51:34 [debug] 38701#0: *112 http check ssl handshake
2013/02/05 13:51:34 [debug] 38701#0: *112 https ssl handshake: 0x16
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL server name: "test.local"
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_do_handshake: -1
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_get_error: 2
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL handshake handler: 0
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_do_handshake: -1
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_get_error: 2
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL handshake handler: 0
2013/02/05 13:51:34 [debug] 38701#0: *112 verify:0, error:20, depth:1, subject:"/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=subca",issuer: "/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=rootca"
2013/02/05 13:51:34 [debug] 38701#0: *112 verify:0, error:27, depth:1, subject:"/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=subca",issuer: "/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=rootca"
2013/02/05 13:51:34 [debug] 38701#0: *112 verify:1, error:27, depth:0, subject:"/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=Client #2",issuer: "/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=subca"
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_do_handshake: 1
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL: TLSv1, cipher: "AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1"
2013/02/05 13:51:34 [debug] 38701#0: *112 http process request line
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_read: 1
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_read: 524
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_read: -1
2013/02/05 13:51:34 [debug] 38701#0: *112 SSL_get_error: 2
2013/02/05 13:51:34 [debug] 38701#0: *112 http request line: "GET / HTTP/1.1"
So I'm getting OpenSSL error #20 and then #27. According to verify documentation:
20 X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY: unable to get local issuer certificate
the issuer certificate could not be found: this occurs if the issuer certificate of an untrusted certificate cannot be found.
27 X509_V_ERR_CERT_UNTRUSTED: certificate not trusted
the root CA is not marked as trusted for the specified purpose.
Any ideas?