I've set up nginx (on Windows) as a front end for Tomcat. Accessing the Tomcat webapp works perfectly when I go directly to Tomcat from either Firefox or IE and when I access it from Java code. It also works perfectly the first time I request it from nginx, but the second time I request it from nginx there is a 60 second delay before nginx responds. Stopping the request in the browser resets the behaviour, so the next response returns immediately and the one following again waits 60 seconds.

On the requests where there is a delay, Nginx sends two upstream requests ("request A" and "request B"). Request A times out after 60 seconds, then request B succeeds. Tomcat does not log request A, so it apparently does not receive it, but responds immediately to request B.

This happens with Nginx's upstream connection set to the default (http 1.0) and also set to use http 1.1 with keepalive 16. It does not appear to be a Tomcat problem because it happens even if I bounce Tomcat between the first and second request (so Tomcat doesn't know it's the second request).

My only other clue is that curl sometimes times out after exactly 150 seconds when accessing the Tomcat url.

I am completely baffled, but it could be something very simple. Any ideas would be appreciated.

Conf file extract (with http/1.0):
server {
<snip>
location / {
proxy_pass http://localhost:8080;
}


Conf file extract (with http/1.1):
upstream tomcat {
server localhost:8080;
keepalive 16;
}

server {
<snip>
location / {
proxy_pass http://tomcat;
proxy_http_version 1.1;
proxy_set_header Connection "";
}

I´m having same issue with a backend with nodejs. My configuration is pretty similar and have also tested changing connection type with no luck (missing requests on my backend and a timeout of 60 seconds until the request is resent).
<snip>
upstream node_app {
server localhost:3000 fail_timeout=5s;
keepalive 1;
}
</snip>

<snip>
location /api {
proxy_http_version 1.1;
proxy_set_header Connection "keep-alive";
proxy_pass http://node_app;
proxy_set_header X-Real-IP $remote_addr;
}
</snip>

On the log I´m getting the following error:
2014/01/30 10:32:32 [error] 6760#3604: *1 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /api/xxxxx HTTP/1.1", upstream: "http://[::1]:3000/api/xxxxx", host: "localhost"


I have been monitoring the ports open with netstat -n and the port 3000, nginx is apparently using a closed socket (port) to send the second request (that is lost and after a minute is re-sent on a new socket/port)

I attach the debug log of two requests



Edited 1 time(s). Last edit at 01/30/2014 08:57AM by Cruz Fernandez.

Can you reach the backend directly without issues? anything in the backend logs?

Hi itpp2012,

The backend always returns fast and healthy. I have debugged the port numbers, and the socket has been closed from backend side.

itpp2012,

Let me rephrase, the first request on the backend appears and closes fast and everything ok. The second request connects to nginx, and backend doesn´t show any connections/requests. After 1 minute, backend gets a connection/request and nginx returns the response.

Thanks for the fast response

HI all,

I am patching this problem with a small timeout parameter (as my nodejs server connects and responds fast, I shouldn't have any issues)

proxy_connect_timeout 2s;

Hopefully this will help while the problem is solved/diagnosed.

Cruz

I use many tomcat backends with nginx as frontend balancing proxy and have not seen this anywhere, if possible could you create a small VM linux distro with nginx to see what happens with open ports between nginx and the backends? All I can see in your logs is a timeout to the backend, this can be caused by anything, wrong expected http version, specific tcp parameters, etc..... Also try the older versions such as 1.4 and 1.2

Thanks itpp2012,

On Linux machine this problem is not happening (we have same configuration on windows and linux). Sorry if it wasn't clear before.

This problem is only happening on Windows machines. I have tried version 1.4.4 with same results. On windows XP 32 bits the problem is not happening, but on Windows 7 64 bits, and Windows Server 2008 64 bits.

Is it possible to create a small VM of windows? Is that what you need to reproduce the error?

Cruz

Ah xp no problem but w7 the problem, this is most likely a ASLR issue, are you using limit_req_zone or other options that used shared memory ? shared memory don't work beyond XP, either keep nginx on XP or don't use shared memory functions until this is fixed, a proper and permanent fix is on the way for my nginx for windows version.

I think i'm not using any parameter that uses shared memory. I am using the standard nginx.conf (the version that comes with windows distribution) and a site.conf file. Attached both config files

You could try sendfile off;

itpp2012,

Looks like nginx version 1.2.9 didn´t have the bug (for windows 7) that is breaking my proxy_pass directive!! Here calling a lot of times the proxy_pass doesn´t break the connection.

On another testing the "sendfile" directive you mentioned, didn´t produced any change of effects

Thanks a lot! The solution of 1.2.9 seems the more productive from my point of view.
Cruz

Hi itpp2012

I have reported what I think the bug is on trac:

http://trac.nginx.org/nginx/ticket/496#ticket

Hope it is clear.

Cheers

Cruz

If your willing you could try some of the releases here http://nginx-win.ecsds.eu/
1.5.10.1 will be out soon, but 1.5.9.1 should not display the behavior your having.

Thanks for your response itpp2012. I've found a way another two workarounds until your changes are merged into a release:

* you can use 127.0.0.1 (instead of localhost on the proxy_pass directive)
or
* you can start the backend on an ipv6 localhost direction (like ::1 )

This workaround works on all nginx version with the problem (1.4.x and until 1.5.9)

I'll test your fixes on http://nginx-win.ecsds.eu/ soon, if that helps!

Cruz

itpp2012,

The release you gave me (1.5.9.1 cheshire), still has the issue of 60 seconds delay, and an error of upstream on "::1" ipv6 address.

(of course, the error happens on windows 2008, with proxy_pass localhost:3000 , and our backend only listening on ipv4 0.0.0.0:3000)

Cruz

The delay sounds like a dns issue, never seen it happen though, I'll put it on the testlist.
About the v6 error, I need the errorlogging and config used to reproduce the issue.
"http://[::1]:3000/api/xxxxx" <- is this properly formatted?

Itpp2012

The nginx conf is the standard one:

<code>
location /api {
proxy_pass http://localhost:3000;
proxy_set_header X-Real-IP $remote_addr;
}
</code>

looks like to reproduce this you might have to change something in the TCP/IP configuration of the machine. I cannot find where is it configured on the localhost loopback on windows, but I can assure that making a ping localhost looks like this:

$ ping localhost

Pinging WIN-XXXXX2882NAME [::1] with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

Didn't maxim explain it was a config mistake? which also includes the v6 error, if you have a separate v6 issue I'd like to see that specific errorlog.

Try adding to your server block:

large_client_header_buffers 4 64k;

This solved it for me.

itpp2012,

i believe my issue is not separate from the ipv6 issue.

In any case the errorlog for the ipv6 problem is the same:
2014/01/30 10:32:32 [error] 6760#3604: *1 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /api/xxxxx HTTP/1.1", upstream: "http://[::1]:3000/api/xxxxx", host: "localhost"

the url http://[::1]:3000/api/xxxxx is the correct URL if we call localhost throught the ipv6 interface.

This might give some more insight: http://mgalgs.github.io/2013/05/04/localhost-considered-harmful.html

I'm using nginx 1.7.3 for windows and it is happening exactly the same issue.

Here are my details:

http://stackoverflow.com/questions/25101451/nodejs-api-and-nginx-proxy-pass-proxy-connect-timeout-reached

That post also mentions a 'fix', is this fix working or are you still having this problem ?

---
nginx for Windows http://nginx-win.ecsds.eu/

One year old but I wanted to point out that there is a work around for that issue. Like Cruz Fernandez wrote you can set 127.0.0.1 instead of localhost on the proxy_pass directive. This prevents the 60s delay on every second request. I'm using Windows 8.1 and nginx 1.9.5.

Cruz Fernandez Wrote:
> * you can use 127.0.0.1 (instead of localhost on the proxy_pass directive)