[cross-posted from https://community.oracle.com/thread/3919426?sr=stream&ru=232867, no reply there]

For an Oracle grown Application (it's actually beehive collab server) that is based on Oracle HTTP Server 10.1.3 (and, being on extended support, cannot be updated to more recent versions of OHS), we have the issue that a fronted NGINX reverse proxy has an issue with with one service (it's actually the server counterpart of the OBEO connector) that is run by a process ombd on Linux, so it appears to be the Oracle Message Broker Daemon:

This one service (only!) appears to be unwilling to accept NGINX proxied connections, and will report "nzos_Handshake failed" in its log files, e.g.

...
xsi:type="nsio-ssl-log"
timestamp="2016-04-05T12:57:54.412Z"
severity="severe"
message="nzssl(context 084163E4): nzos_Handshake failed, error: 28860" />
</log>
xsi:type="nsio-ssl-log"
timestamp="2016-04-05T12:57:54.412Z"
severity="severe"
message="nzssl(context 084163E4): nzos_Handshake failed, error: 28860" />
</log>

The various documents on MOS regarding nzos_Handshake failed, error: 28860 did not help.

The NGINX-Proxy, run with debug logging will report "SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number" at the same moment:

...
2016/04/05 14:51:08 [debug] 13091#0: *19 http upstream request: "/beehive/redirect/secure-mx?"
2016/04/05 14:51:08 [debug] 13091#0: *19 http upstream process upstream
2016/04/05 14:51:08 [debug] 13091#0: *19 pipe read upstream: 1
2016/04/05 14:51:08 [debug] 13091#0: *19 SSL_read: -1
2016/04/05 14:51:08 [debug] 13091#0: *19 SSL_get_error: 1
2016/04/05 14:51:08 [error] 13091#0: *19 SSL_read() failed (SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number) while reading upstream, client: 172.31.23.5, server: beehive.site.com, request: "GET /beehive/redirect/secure-mx HTTP/1.0", upstream: "https://myip:myport/beehive/redirect/secure-mx" [^]
2016/04/05 14:51:08 [debug] 13091#0: *19 pipe recv chain: -1
2016/04/05 14:51:08 [debug] 13091#0: *19 event timer del: 34: 1459861268505
2016/04/05 14:51:08 [debug] 13091#0: *19 http upstream exit: 0000000000000000
2016/04/05 14:51:08 [debug] 13091#0: *19 finalize http upstream request: 502
2016/04/05 14:51:08 [debug] 13091#0: *19 finalize http proxy request
2016/04/05 14:51:08 [debug] 13091#0: *19 free rr peer 1 0
2016/04/05 14:51:08 [debug] 13091#0: *19 SSL_shutdown: 1
...

Wireshark reports "Alert (Level: Fatal, Description: Protocol Version)".

We do not want to front BigIP.

I believe we are seeing some incompatibility between Application Server using SSL-libraries from Certicom and the OpenSSL Libraries used in Linux.

Can somebody offer an idea on how to get this running: As a matter of fact, Oracle HTTP Server 10.1.3 will not accept SHA-1 certificates, and the current certificate will terminate in a few months.

Kind regards, Tom

What about this;
http://docs.oracle.com/cd/E28280_01/web.1111/e13707/ssl.htm#SECMG384

However any device which still uses sslv3 (and can't use sslv2), uses expired certificates which can't be updated should be accessed via plain http and proxied by a secure endpoint (such as nginx), if such (upstream) link is not considered secure enough tunnel this via vpn/ipsec or a simple stunnel.

---
nginx for Windows http://nginx-win.ecsds.eu/

WebLogic Server 10.3.6 is not the same as OHS (Oracle HTTP Server) 10.1.3, which is an obsolete product in this version. The OHS is TLS enabled, but appears to have issues when talking with the proxy NGINX.

Functionality is different yes but the technical way connections are made are nearly the same for all oracle webservers, it is a ssl3/ssl2 issue.

Consider an end2end (upstream) vlan [1] between nginx and oracle and talk http over the upstream. SSL terminate at nginx.

[1] If a backend (upstream) can't use proper encryption you might as well not use it at all.

---
nginx for Windows http://nginx-win.ecsds.eu/

Are you really sure it is a ssl3/ssl2 issue? When I do a trace where NGINX is not involved (direct communication between the OBEO plugin and OHS), wireshark reports ssl.record.version Version: TLS 1.0 (0x0301), so SSL3 should not be talked here, as 0x0301 is aka TLS 1.0.

swadm Wrote:
-------------------------------------------------------
> Are you really sure it is a ssl3/ssl2 issue?

"SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"

http://stackoverflow.com/questions/29610075/ssl-error-error1408f10bssl-routinesssl3-get-recordwrong-version-number-erro
https://community.oracle.com/thread/3870862?start=0&tstart=0

Could be someone did some POODLE mitigation but forgot to 'tell' nginx (configuration).

---
nginx for Windows http://nginx-win.ecsds.eu/

[Thanks for the continued effort in assisting!]

I know that OHS in the beehive flavour is limited to TLSv1, and cannot be updated to support newer versions and also no SHA-256 certificates. That's actually why we needed to front NGINX to get various browsers to accept the connection.

All of these connections work fine, but only the OBEO component will refuse a successful connection.

As far as I understand, a redirector process will hand over the initial connect to another process, the abovementioned ombd: I'm seeing another Server Hello and "TLSv1 Record Layer: Change Cipher Spec" after the HTTP GET "/beehive/redirect/secure-mx".

Perhaps it is the other way round: ombd may be the only component missing poodle mitigation. How would I tell NGINX: it did not help to include SSLv3 in the proxy_ssl_protocols parameter.

It would be better, by going through debug logging, to see (map) who is talking to who, who is redirecting and where to and what is handling the redirected request.

I think I've seen something in the latest nginx(openssl) changesets about removing ssl3.
With openssl you may need to explicitly include ssl3.

When it comes to re-directors, they often need to be told (with a proxy) a different end-point is used or they will send a 'return' url pointing to themselves.

---
nginx for Windows http://nginx-win.ecsds.eu/

Just to be sure of a common understanding: with handover by the
redirector I mean that, while only httpd is listening on the server
port, the same port is having ESTABLISHED sessions with different
programs (java, httpd, ombd), so my understanding is that the NGINX
configuration is able to talk to httpd and the java processes, but
failing with ombd.

I would like to get an even better understanding of the communication
of what is being talked between NGINX as proxy and the server, but I
am still seeing "Encrypted Application Data" on that part of the road,
although I successfully imported the private keys of both NGINX and
the server.

When NGINX is talking to the server, what I see on TLS layer, is the
following:

TLSv1 Record Layer: Handshake Protocol: Client Hello
TLSv1 Record Layer: Handshake Protocol: Client Hello
TLSv1 Record Layer: Handshake Protocol: Server Hello
TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
TLSv1 Record Layer: Handshake Protocol: Server Hello
TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
TLSv1 Record Layer: Application Data Protocol: http
TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
TLSv1 Record Layer: Application Data Protocol: http
TLSv1 Record Layer: Application Data Protocol: http
TLSv1 Record Layer: Application Data Protocol: http
TLSv1 Record Layer: Encrypted Alert
TLSv1 Record Layer: Alert (Level: Fatal, Description: Close Notify)
TLSv1 Record Layer: Encrypted Alert
TLSv1 Record Layer: Alert (Level: Fatal, Description: Close Notify)

... but all application data is still encrypted.

The four "Change Cipher Spec" in rapid successions are not healthy, I
deem.

swadm Wrote:
-------------------------------------------------------
> The four "Change Cipher Spec" in rapid successions are not healthy, I
> deem.

Without knowing the details you can't say one or the other. Once you know details such as error messages or other more detailed descriptions you can use google.

Connect to the backend with Curl or firefox in development mode and see which cypher-list is being used and compare that with what nginx wants to use (or is being forced not to use).

---
nginx for Windows http://nginx-win.ecsds.eu/

Tested with curl:

In the trace file written with the --trace parameter, I find that, without NGINX, the GET request is simply served, and then reports "Info: Connection #0 to host ... left intact".

In the case of NGINX inbetween, I get a

HTTP/1.1 302 Moved Temporarily..

According to the trace file, Curl will then request "Location: ...", and receive "302 Found", then the trace file ends with "Info: Closing connection 0" is closed.

Is it OK, that NGINX proxy involves a "HTTP/1.1 302 Moved Temporarily" message?

Regards, Thomas

Then you must have some return/rewrite doing a 302 in your nginx configuration. If this is intended then its ok.

---
nginx for Windows http://nginx-win.ecsds.eu/

I checked, but no, there is no return or rewrite in the NGINX configuration.

Sorry, but I used http instead of https, so the last test was useless.

Another go ... now the curl picture is as follows: the tracefiles of
CURL doing ...

curl https://my-server.my-domain:55094/beehive/redirect/secure-mx --tlsv1.0 --stderr curl-err.txt --output curl-out.txt --trace curl-trace.txt

... look quite similar (we know that Oracle 10.1.3 will only speak
TLSV1.0), and I don't see any TLS negotiation issues ... strange!

I must be the special combination:

curl <-> nginx <-> OHS

appears to work (although fetching the single page via curl is a very
limited test), and

OBEO <-> nginx <-> OHS

will fail (OBEO are the Outlook extensions to work with the beehive
server).

I can see that OBEO sends a quite restricted choice of Cipher Suites
when doing the "Client hello":

Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_SHA (0x0064)
Cipher Suite: TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA (0x0062)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC4_40_MD5 (0x0003)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 (0x0006)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_DHE_DSS_WITH_DES_CBC_SHA (0x0012)
Cipher Suite: TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA (0x0063)

without NGINX, OHS will, according to Server hello, choose

Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)

with NGINX, the Server hello of NGINX indicates

Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)

and further on, the OHS communication with NGINX will settle on

Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)

It is still my suspect that the handover to the abovementioned ombd
incurs the failing when NGINX is in the chain, resulting in the debug
message ...

SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

... that I cited above.

Try a forced setting of only TLS_RSA_WITH_RC4_128_SHA for nginx front and backend.

---
nginx for Windows http://nginx-win.ecsds.eu/

[thanks for the continued timely assistance - great!]

tried

ssl_ciphers TLS_RSA_WITH_RC4_128_SHA;

and, simultaneousely

proxy_ssl_ciphers TLS_RSA_WITH_RC4_128_SHA;

but, on both SLES11 and SLES12, nginx reports

nginx: [emerg] SSL_CTX_set_cipher_list("TLS_RSA_WITH_RC4_128_SHA") failed (SSL: error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match)

Is it correct to assume that the ciphers need to come from the SSL libs of the OS, and cannot be "configured" to include a matching cipher like TLS_RSA_WITH_RC4_128_SHA?



Edited 1 time(s). Last edit at 04/29/2016 08:24AM by swadm.

Could be, the idea was once a cipher is forced it either works or you get some other error pointing to the real problem.

However you need to translate that cipher value into what nginx configuration understands;

See also
https://blog.qualys.com/ssllabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy
http://security.stackexchange.com/questions/87326/what-is-the-impact-of-removing-tls-rsa-with-rc4-128-sha-from-my-servers-cipher

---
nginx for Windows http://nginx-win.ecsds.eu/

NGINX wants to have a cipher, that matches the output of "openssl ciphers", right?

With "openssl ciphers -V" (which outputs the binary value of the Cipher, too), I found that TLS_RSA_WITH_RC4_128_SHA (0x0005) equals

0x00,0x05 - RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1

Put this into nginx.conf on both upstream and downstream, restarted, and retried.

I see that all server hellos confirm the TLS_RSA_WITH_RC4_128_SHA cipher, still the communication will abort shortly after
"Alert Message: Encrypted Alert", again with "SSL_read() failed (SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number) while reading upstream" in the nginx log file.

By now I'm quite pessimistic that I'll ever find a solution for this setup!

You can try the mailing list at https://forum.nginx.org/list.php?2
or the openssl forum on google groups, a detailed wireshark capture might be helpful for some to figure out what is going on.

---
nginx for Windows http://nginx-win.ecsds.eu/