Hi.

I want to use nginx as a load balancer for tcp streams and because I want to share the load evenly over all machines I thought that least_conn would be the right strategy to use in this case. But I have stumbled upon an problem I don't really understand.

In my lab environment I have three machines running server software.

I start my testing establishing 12 connections to nginx and can see that they are getting evenly across all hosts. 4 connections per host.
I then stop one of the machines and the client reestablishes the lost connections and they are getting spread evenly to the remaining hosts. Now we have 6 connections on 2 hosts.
I then start up the closed machine and proceed to close 2 connections each on the servers that have been up all this time. The client tries to set up new connections and this is where I notice that the load not being equally shared anymore.
The results I get is host1: 6 connections, host2: 5 connections, host3: 1 connection.

As far as I can see it seems like nginx uses round-robin to balance the connections and not setting up connections to the host with least connections. Am I misinterpreting what least connections mean?

From the documentation

Syntax: least_conn;
Default: —
Context: upstream
Specifies that a group should use a load balancing method where a connection is passed to the server with the least number of active connections, taking into account weights of servers. If there are several such servers, they are tried in turn using a weighted round-robin balancing method.

How I interpret this is that it will only use weighted round-robin in the case where there are 2 or more servers with the same amount connections that satisfy the condition of having the least number of connections, which should not be the case in this where host3 should be the one with least amount of connections.

I had the hypothesis that I used a shared memory zone in the upstream might be the cause of the problem, but removing that only made the results even more weird.

After removing the zone the initial step of setting up 12 connections got me the following results
Host1: 6 connections, Host2: 5 connections, Host3: 1 connection.

I suspect that it is the core config that might be faulty but tweaking the parameters don't seem to get me further to solve this.

My configuration

user nginx;
worker_processes 16;
error_log /opt/nginx/logs/errors.log;
worker_rlimit_nofile 10000;
events {
worker_connections 2048;
worker_aio_requests 8;
}

stream {
upstream stream1 {
least_conn;
zone stream1 128k;
server host3:19953;
server host2:19953;
server host1:19953;
}
server {
listen 9953;
proxy_pass stream1;
}

access_log off;
}