Welcome! Log In Create A New Profile

Advanced

Upstream on Windows 60 second delay

Posted by n_2 
n_2
Upstream on Windows 60 second delay
June 01, 2013 09:27PM
I've set up nginx (on Windows) as a front end for Tomcat. Accessing the Tomcat webapp works perfectly when I go directly to Tomcat from either Firefox or IE and when I access it from Java code. It also works perfectly the first time I request it from nginx, but the second time I request it from nginx there is a 60 second delay before nginx responds. Stopping the request in the browser resets the behaviour, so the next response returns immediately and the one following again waits 60 seconds.

On the requests where there is a delay, Nginx sends two upstream requests ("request A" and "request B"). Request A times out after 60 seconds, then request B succeeds. Tomcat does not log request A, so it apparently does not receive it, but responds immediately to request B.

This happens with Nginx's upstream connection set to the default (http 1.0) and also set to use http 1.1 with keepalive 16. It does not appear to be a Tomcat problem because it happens even if I bounce Tomcat between the first and second request (so Tomcat doesn't know it's the second request).

My only other clue is that curl sometimes times out after exactly 150 seconds when accessing the Tomcat url.

I am completely baffled, but it could be something very simple. Any ideas would be appreciated.

Conf file extract (with http/1.0):
server {
<snip>
location / {
proxy_pass http://localhost:8080;
}


Conf file extract (with http/1.1):
upstream tomcat {
server localhost:8080;
keepalive 16;
}

server {
<snip>
location / {
proxy_pass http://tomcat;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
Re: Upstream on Windows 60 second delay
January 30, 2014 08:56AM
I´m having same issue with a backend with nodejs. My configuration is pretty similar and have also tested changing connection type with no luck (missing requests on my backend and a timeout of 60 seconds until the request is resent).
<snip>
upstream node_app {
server localhost:3000 fail_timeout=5s;
keepalive 1;
}
</snip>

<snip>
location /api {
proxy_http_version 1.1;
proxy_set_header Connection "keep-alive";
proxy_pass http://node_app;
proxy_set_header X-Real-IP $remote_addr;
}
</snip>

On the log I´m getting the following error:
2014/01/30 10:32:32 [error] 6760#3604: *1 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /api/xxxxx HTTP/1.1", upstream: "http://[::1]:3000/api/xxxxx", host: "localhost"


I have been monitoring the ports open with netstat -n and the port 3000, nginx is apparently using a closed socket (port) to send the second request (that is lost and after a minute is re-sent on a new socket/port)

I attach the debug log of two requests



Edited 1 time(s). Last edit at 01/30/2014 08:57AM by Cruz Fernandez.
Attachments:
open | download - error.log (128.8 KB)
Re: Upstream on Windows 60 second delay
January 30, 2014 09:10AM
Can you reach the backend directly without issues? anything in the backend logs?
Re: Upstream on Windows 60 second delay
January 30, 2014 09:35AM
Hi itpp2012,

The backend always returns fast and healthy. I have debugged the port numbers, and the socket has been closed from backend side.
Re: Upstream on Windows 60 second delay
January 30, 2014 09:47AM
itpp2012,

Let me rephrase, the first request on the backend appears and closes fast and everything ok. The second request connects to nginx, and backend doesn´t show any connections/requests. After 1 minute, backend gets a connection/request and nginx returns the response.

Thanks for the fast response
Re: Upstream on Windows 60 second delay
January 30, 2014 11:40AM
HI all,

I am patching this problem with a small timeout parameter (as my nodejs server connects and responds fast, I shouldn't have any issues)

proxy_connect_timeout 2s;

Hopefully this will help while the problem is solved/diagnosed.

Cruz
Re: Upstream on Windows 60 second delay
January 30, 2014 12:49PM
I use many tomcat backends with nginx as frontend balancing proxy and have not seen this anywhere, if possible could you create a small VM linux distro with nginx to see what happens with open ports between nginx and the backends? All I can see in your logs is a timeout to the backend, this can be caused by anything, wrong expected http version, specific tcp parameters, etc..... Also try the older versions such as 1.4 and 1.2
Re: Upstream on Windows 60 second delay
January 30, 2014 01:12PM
Thanks itpp2012,

On Linux machine this problem is not happening (we have same configuration on windows and linux). Sorry if it wasn't clear before.

This problem is only happening on Windows machines. I have tried version 1.4.4 with same results. On windows XP 32 bits the problem is not happening, but on Windows 7 64 bits, and Windows Server 2008 64 bits.

Is it possible to create a small VM of windows? Is that what you need to reproduce the error?

Cruz
Re: Upstream on Windows 60 second delay
January 30, 2014 02:13PM
Ah xp no problem but w7 the problem, this is most likely a ASLR issue, are you using limit_req_zone or other options that used shared memory ? shared memory don't work beyond XP, either keep nginx on XP or don't use shared memory functions until this is fixed, a proper and permanent fix is on the way for my nginx for windows version.
Re: Upstream on Windows 60 second delay
January 30, 2014 03:06PM
I think i'm not using any parameter that uses shared memory. I am using the standard nginx.conf (the version that comes with windows distribution) and a site.conf file. Attached both config files
Attachments:
open | download - design_dashboard.conf (578 bytes)
open | download - nginx.conf (2.7 KB)
Re: Upstream on Windows 60 second delay
January 30, 2014 03:16PM
You could try sendfile off;
Re: Upstream on Windows 60 second delay
January 30, 2014 03:50PM
itpp2012,

Looks like nginx version 1.2.9 didn´t have the bug (for windows 7) that is breaking my proxy_pass directive!! Here calling a lot of times the proxy_pass doesn´t break the connection.

On another testing the "sendfile" directive you mentioned, didn´t produced any change of effects

Thanks a lot! The solution of 1.2.9 seems the more productive from my point of view.
Cruz
Re: Upstream on Windows 60 second delay
February 03, 2014 10:25AM
Hi itpp2012

I have reported what I think the bug is on trac:

http://trac.nginx.org/nginx/ticket/496#ticket

Hope it is clear.

Cheers

Cruz
Re: Upstream on Windows 60 second delay
February 03, 2014 12:46PM
If your willing you could try some of the releases here http://nginx-win.ecsds.eu/
1.5.10.1 will be out soon, but 1.5.9.1 should not display the behavior your having.
Re: Upstream on Windows 60 second delay
February 04, 2014 09:09AM
Thanks for your response itpp2012. I've found a way another two workarounds until your changes are merged into a release:

* you can use 127.0.0.1 (instead of localhost on the proxy_pass directive)
or
* you can start the backend on an ipv6 localhost direction (like ::1 )

This workaround works on all nginx version with the problem (1.4.x and until 1.5.9)

I'll test your fixes on http://nginx-win.ecsds.eu/ soon, if that helps!

Cruz
Re: Upstream on Windows 60 second delay
February 04, 2014 09:59AM
itpp2012,

The release you gave me (1.5.9.1 cheshire), still has the issue of 60 seconds delay, and an error of upstream on "::1" ipv6 address.

(of course, the error happens on windows 2008, with proxy_pass localhost:3000 , and our backend only listening on ipv4 0.0.0.0:3000)

Cruz
Re: Upstream on Windows 60 second delay
February 04, 2014 01:06PM
The delay sounds like a dns issue, never seen it happen though, I'll put it on the testlist.
About the v6 error, I need the errorlogging and config used to reproduce the issue.
"http://[::1]:3000/api/xxxxx" <- is this properly formatted?
Re: Upstream on Windows 60 second delay
February 04, 2014 01:29PM
Itpp2012

The nginx conf is the standard one:

<code>
location /api {
proxy_pass http://localhost:3000;
proxy_set_header X-Real-IP $remote_addr;
}
</code>

looks like to reproduce this you might have to change something in the TCP/IP configuration of the machine. I cannot find where is it configured on the localhost loopback on windows, but I can assure that making a ping localhost looks like this:

$ ping localhost

Pinging WIN-XXXXX2882NAME [::1] with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Re: Upstream on Windows 60 second delay
February 04, 2014 01:40PM
Didn't maxim explain it was a config mistake? which also includes the v6 error, if you have a separate v6 issue I'd like to see that specific errorlog.
Re: Upstream on Windows 60 second delay
February 04, 2014 01:40PM
Try adding to your server block:

large_client_header_buffers 4 64k;

This solved it for me.
Re: Upstream on Windows 60 second delay
February 04, 2014 03:33PM
itpp2012,

i believe my issue is not separate from the ipv6 issue.

In any case the errorlog for the ipv6 problem is the same:
2014/01/30 10:32:32 [error] 6760#3604: *1 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /api/xxxxx HTTP/1.1", upstream: "http://[::1]:3000/api/xxxxx", host: "localhost"

the url http://[::1]:3000/api/xxxxx is the correct URL if we call localhost throught the ipv6 interface.
Re: Upstream on Windows 60 second delay
February 04, 2014 03:49PM
This might give some more insight: http://mgalgs.github.io/2013/05/04/localhost-considered-harmful.html
Re: Upstream on Windows 60 second delay
August 03, 2014 02:23AM
I'm using nginx 1.7.3 for windows and it is happening exactly the same issue.

Here are my details:

http://stackoverflow.com/questions/25101451/nodejs-api-and-nginx-proxy-pass-proxy-connect-timeout-reached
Re: Upstream on Windows 60 second delay
August 03, 2014 05:28AM
That post also mentions a 'fix', is this fix working or are you still having this problem ?

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: Upstream on Windows 60 second delay
October 13, 2015 04:44AM
One year old but I wanted to point out that there is a work around for that issue. Like Cruz Fernandez wrote you can set 127.0.0.1 instead of localhost on the proxy_pass directive. This prevents the 60s delay on every second request. I'm using Windows 8.1 and nginx 1.9.5.

Cruz Fernandez Wrote:
> * you can use 127.0.0.1 (instead of localhost on the proxy_pass directive)
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 80
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready