I am using nginx to distribute http load across three upstream application servers. It actually works really well in normal usage, but when we restart one of the upstream servers for maintenance (such as for an OS update), nginx tends to hang incoming requests for some time (30 seconds or more). When a node is brought offline, the nginx error logs show a few messages like this: upstream prematurely closed connection while reading response header from upstream Followed by many messages like this (these last basically as long as the node is offline): kevent() reported that connect() failed (61: Connection refused) while connecting to upstream This seems ok to me, since the upstream server did go away. What should I expect to see of incoming http requests when this happens? Shouldn't nginx route requests to the remaining application servers, making the outage invisible to users? If it's expected to hang some requests when this happens, what timeout could be adjusted to minimize that duration? Thanks, - .Dustin _______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginxhttp://forum.nginx.org/read.php?2,227729,227729#msg-227729Tue, 21 May 2013 23:04:22 -0400 Phorum 5.2.16 http://forum.nginx.org/read.php?2,227729,227737#msg-227737 http://forum.nginx.org/read.php?2,227729,227737#msg-227737> small value if responsiveness during an outage is desired?

It depends on the nature of the (backend) software and the network
infrastucture/latency - the closer the servers (network-wise) and the
higher throughput of the application the lower you can/should set it.

There is also a general observation/standard or apdex (Application
Performance Index) that average web-user starts to become "unsatisfied" if
the request takes more than 1-2 seconds so using the default 60 seconds (in
case of some backend failure) to deliver the content makes no sense - the
user will sooner hit refresh / just close the browsers (go to another site)
or fall in panic than wait for anything to load.


On the other hand making nginx immideatly close and switch to another
backend isnt always the (best) solution - especially if the backends do some
long running computations and/or accept only limited number of connections.
This way quickly cycling through all backends won't serve a good response at
all what matters if you have an extra caching layer above (to fetch a valid
object (html/image etc)) or some sort of transactions (like payment systems)
which just take time and each request is important.


I usually set it to 2-3 seconds.
That way I can also indicate if the backends perform well enough and react
if a new bottleneck arises - the application code, underlaying
DBs/filesystems start to get too slow.


rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx]]> Reinis Rozitis Nginx Mailing List - EnglishWed, 20 Jun 2012 17:00:00 -0400 http://forum.nginx.org/read.php?2,227729,227736#msg-227736 http://forum.nginx.org/read.php?2,227729,227736#msg-227736
upstream pool012 {
server 172.16.1.223:80;
server 172.16.1.224:80;
server 172.16.1.225:80;
server 172.16.1.226:80;
server 172.16.1.227:80;
}

keepalive_timeout 65;

server {
listen 172.16.6.103:80;
server_name application012.domain.com;

error_log logs/application012.error.log ;

location / {
proxy_pass http://pool012;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
proxy_redirect off;
proxy_read_timeout 930s;
}
}


Is it considered good practice to reduce proxy_connect_timeout to some small value if responsiveness during an outage is desired?

- .Dustin

On Jun 20, 2012, at 2:04 PM, Reinis Rozitis wrote:

>> I am using nginx to distribute http load across three upstream application servers. It actually works really well in normal usage, but when we restart one of the upstream servers for maintenance (such as for an OS update), nginx tends to hang incoming requests for some time (30 seconds or more). When a node is brought offline, the nginx error logs show a few messages like this:
>
>
> You should probably post the relevant parts of your configuration since there are quite few parameters to tune nginx for a better responsiveness.
>
> To name a few: proxy_connect_timeout, proxy_read_timeout, proxy_send_timeout which by default are 60 seconds (what could explain your "hanging" requests), lowering those allows 'proxy_next_upstream' (by default - timeout and error) kick in sooner so the backend changes/restarts are seemless and don't really affect the end-users so much.
>
> rr
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx]]> Dustin Wenz Nginx Mailing List - EnglishWed, 20 Jun 2012 15:42:01 -0400 http://forum.nginx.org/read.php?2,227729,227735#msg-227735 http://forum.nginx.org/read.php?2,227729,227735#msg-227735> servers. It actually works really well in normal usage, but when we
> restart one of the upstream servers for maintenance (such as for an OS
> update), nginx tends to hang incoming requests for some time (30 seconds
> or more). When a node is brought offline, the nginx error logs show a few
> messages like this:


You should probably post the relevant parts of your configuration since
there are quite few parameters to tune nginx for a better responsiveness.

To name a few: proxy_connect_timeout, proxy_read_timeout, proxy_send_timeout
which by default are 60 seconds (what could explain your "hanging"
requests), lowering those allows 'proxy_next_upstream' (by default - timeout
and error) kick in sooner so the backend changes/restarts are seemless and
don't really affect the end-users so much.

rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx]]> Reinis Rozitis Nginx Mailing List - EnglishWed, 20 Jun 2012 15:06:01 -0400 http://forum.nginx.org/read.php?2,227729,227729#msg-227729 http://forum.nginx.org/read.php?2,227729,227729#msg-227729
upstream prematurely closed connection while reading response header from upstream

Followed by many messages like this (these last basically as long as the node is offline):

kevent() reported that connect() failed (61: Connection refused) while connecting to upstream

This seems ok to me, since the upstream server did go away. What should I expect to see of incoming http requests when this happens? Shouldn't nginx route requests to the remaining application servers, making the outage invisible to users? If it's expected to hang some requests when this happens, what timeout could be adjusted to minimize that duration?

Thanks,

- .Dustin
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx]]> Dustin Wenz Nginx Mailing List - EnglishWed, 20 Jun 2012 13:28:00 -0400