I had to start looking at this issue again now that yet another openssl security issue. Now that I know I can go back to a working setup just by downgrading SSL, I am able to gather more information.

This morning, I updated the libssl libraries and restarted nginx, and the errors started flooding back. This time, I took a packet capture to see what was happening and what I could correlate. I run a set of servers that handle API requests from a mobile phone application, and every single client that produced this error was running iOS.

In the packet capture, we offer the same cipher that the clients always use without a problem, but for some reason, some of our iPhone clients have issues (not all.) I have been unable to discern a pattern, but it's always iPhones and doesn't seem to have anything to do with the device model or the OS version. I haven't found a single Android instance of the IP's that show up in our error logs, and we have slightly more Android devices than iOS devices.

We get the Client Hello which has a list of 37 potential ciphers for TLS 1.2. We send the server hello and offer the normal cipher. The client, instead of continuing on, immediately sends a FIN, ACK. It then tries to connect again over TLS 1.0, gives the client hello, we send the ACK and almost immediately, WE send a FIN, ACK to the client.

Since it's an API and there are multiple requests being made from the client, not every one will fail. Some negotiate SSL just fine, others do not.

I'm still digging through the packet captures to try and figure out any other patterns.

As soon as I downgrade libssl, everything works fine.