Yes, I am able to reproduce this talking to the same nginx as an upstream,
here is my new config. To reproduce, create a file in the root which is
several Mb, i used 20Mb, and issus multiple simultaneous curl's to the
object, i found rate-limiting my curl is the best way to repro. This
suggests there is some problem when we have to buffer. I'm skeptical that
this is an openssl issue as I have used multiple different openssl versions
and still run into this. However for completeness, I've reprod with
openssl sources from openssl.org (openssl-1.0.1e) as you suggested.
Updated conf:
### Begin ngxin.conf ###
worker_processes 1;
error_log logs/error.log debug;
pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
access_log logs/access.log;
keepalive_timeout 60;
upstream http {
server 127.0.0.1:1183;
keepalive 512;
}
server {
listen 1182 default_server;
server_name -;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://http;
proxy_redirect off;
proxy_read_timeout 10s;
proxy_connect_timeout 6s;
proxy_buffering off;
proxy_buffer_size 64k;
proxy_buffers 6 16k;
proxy_busy_buffers_size 80k;
proxy_pass_header Server;
proxy_pass_header Date;
proxy_pass_header X-Pad;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Host "upstream.srv";
}
}
server {
listen 1183 ssl;
server_name upstream.srv;
ssl_certificate /var/nginx/conf/upstream.srv.pem;
ssl_certificate_key /var/nginx/conf/upstream.srv.key;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
root /var/nginx/html;
}
}
}
### End ngxin.conf ###
configure flags for nginx 1.4.3:
../configure --prefix=/var/nginx --with-debug --with-http_ssl_module
--without-http_auth_basic_module --without-http_autoindex_module
--without-http_browser_module --without-http-cache
--without-http_charset_module --without-http_empty_gif_module
--without-http_fastcgi_module --without-http_geo_module
--without-http_gzip_module --without-http_limit_conn_module
--without-http_map_module --without-http_memcached_module
--without-http_referer_module --without-http_rewrite_module
--without-http_scgi_module --without-http_split_clients_module
--without-http_ssi_module --without-http_upstream_ip_hash_module
--without-http_userid_module --without-http_uwsgi_module
--without-mail_imap_module --without-mail_pop3_module
--without-mail_smtp_module --with-openssl=/tmp/openssl-1.0.1e
I start nginx and then issue 3 simultaneous curl's from the local box,
rate-limited. This should be sufficient for anyone else to repro the issue.
curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&
curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&
curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&
At least one will fail with bytes remaining, and you will see the error in
the error.log.
2013/10/18 19:56:50 [error] 14667#0: *4 SSL_read() failed (SSL:
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record
mac) while sending to client, client: 127.0.0.1, server: -, request: "GET
/20m.bin HTTP/1.1", upstream: "https://127.0.0.1:1183/20m.bin", host:
"localhost:1182"
thanks for your attention.
On Fri, Oct 18, 2013 at 7:06 PM, Maxim Dounin <mdounin@mdounin.ru> wrote:
> Hello!
>
> On Fri, Oct 18, 2013 at 06:01:14PM +0000, Agent Coulson wrote:
>
> > I am able to reproduce the following error when I have nginx configured
> > with an upstream https connection. I have tweaked various settings all
> to
> > no avail (proxy_buffer_size, proxy_buffers, proxy_ssl_session_reuse).
> >
> > 2013/10/18 17:17:31 [debug] 15644#0: *39 SSL_read: -1, SSL_pending: 16384
> > 2013/10/18 17:17:31 [debug] 15644#0: *39 SSL_get_error: 1
> > 2013/10/18 17:17:31 [error] 15644#0: *39 SSL_read() failed (SSL:
> > error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad
> record
> > mac) while sending to client, client: 127.0.0.1, server: -, request: "GET
> > /test-1 HTTP/1.1", upstream: "https://x.x.x.x:443/test-1", host:
> > "localhost:1182"
>
> I tend to think it's highly unlikely it's a problem in nginx.
> Most likely, it's a problem either in OpenSSL library used on
> nginx side, or in SSL implementation used on a backend.
>
> First thing I would recommend to test is to make sure you are able
> to reporoduce the problem:
>
> 1. Using nginx statically compiled with a known version of the
> OpenSSL library (--with-openssl=..., with sources from
> openssl.org).
>
> 2. Using the same nginx as a backend.
>
> [...]
>
> > I've seen a bug report on this too (
> http://trac.nginx.org/nginx/ticket/215),
> > so thought i would send this here to see if anyone else is actively
> working
> > on the issue.
>
> As of now, no one provided enough steps to reproduce the problem.
> And, see above, most likely the problem is not in nginx.
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
