Maxim Dounin Wrote:
-------------------------------------------------------
> If you see problems with nginx 1.7.9, consider following hints
> at http://wiki.nginx.org/Debugging.
I think it will not help (at least if not did by anyone who really knows both openssl and nginx internals).
the problem is quickly traceable to

long
ssl3_ctx_callback_ctrl(SSL_CTX *ctx, int cmd, void (*fp)(void))
{
CERT *cert;

cert = ctx->cert;

switch (cmd) {
case SSL_CTRL_SET_TMP_RSA_CB:
SSLerr(SSL_F_SSL3_CTX_CTRL, ERR_R_SHOULD_NOT_HAVE_BEEN_CALLED);
(yes, this occurence, exactly)

inside libressl-2.1.3/ssl/s3_lib.c, and this function seems newer called by nginx code directly and not supposed to be externally-called at all.
The pure openssl have some pointer-magic in this place, dropped by libressl developers (with the data structure itself, so no easy way to bring it back)

I think the only thing developers may do (if not willing to really investigate and fix this issue) - just stop declaring nginx compatibility with libressl. It not only nonworking, but worse - it cleanly execute some garbage instead of code.
(I have full system log of stack-protection mechanics trying to prevent this)

and yes, 1.7.10 still does the same. The problem itself does not appear on any connection, just in some special cases, but easely reproduceable.