We've been experiencing a weird issue with nginx used as a gzipping proxy. One of our internal services doesn't support content-encoding and because it usually streams huge amounts of json data in response, we've decided to put it behind nginx. Unfortunately, about 5-10% of responses in this configuration seem to just stall with the TCP connection perfectly fine, but no data going anywhere.
I've tried to diagnose the issue to check if it's a problem with our HTTP libraries, VPN or network, but it does look like nginx is the culprit:
- The issue is reproducible with any HTTP client, including curl --compress.
- Packet capture on the server that's hosting a client and the server that's hosting nginx show nothing interesting. There are no lost or corrupted packets; the client receives the last packet sent by nginx, ACKs it and then ACKs keepalives. If the client is killed, it and nginx exchange FIN/FIN-ACKs just fine.
- Packet capture on the loopback interface, however, shows that nginx just stops reading data at some point (I see a “full window” notice in wireshark and netstat shows an nginx worker sitting with a 512k-2M receive queue with an empty transmit queue).
Some technical details (I've masked our internal host names):
The server on which nginx is on is running a server build of Ubuntu; uname -a gives Linux <hostname> 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux. This issue occurs on nginx from the Ubuntu repo (1.1.19-1ubuntu0.1) and Debian wheezy repo (1.2.1-2.2).
nginx -V output:
nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-pcre-jit --with-debug --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module --add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-auth-pam --add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-echo --add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upstream-fair --add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-dav-ext-module
Basic nginx config is unchanged from Ubuntu-provided defaults; the server section looks like this:
server {
gzip on;
gzip_proxied any;
gzip_types application/json;
listen 80 default;
server_name <servername>;
access_log /var/log/nginx/<servername>-access.log;
location / {
proxy_pass http://127.0.0.1:8083;
proxy_buffering off;
}
}
You can see the nginx debug log here: http://pastebin.com/qDis2RS4 (I've masked the request URL but it's otherwise unchanged); it ends about 1 second after the response stall.
