Welcome! Log In Create A New Profile

Advanced

UDP Load balancer does not scale

May 16, 2017 03:10AM
Hi

I am trying to set up a UDP load balancer using Nginx. Initially, I configured 4 usptream servers with two server processes running on each of them.
It gave a throughput of around 24000 query per second when tested with dnsperf. When I try to add two more upstreams servers, the throughput is not increasing as expected. In fact, it deteriorates to the range of 5000 query per second with the following error:

[warn] 5943#0: *10433175 upstream server temporarily disabled while proxying connection, udp client: xxx.xxx.xxx.29, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.224:53", bytes from/to client:80/0, bytes from/to upstream:0/80
[error] 5943#0: *10085077 no live upstreams while connecting to upstream, udp client: xxx.xxx.xxx.224, server: 0.0.0.0:53, upstream: "dns_upstreams", bytes from/to client:80/0, bytes from/to upstream:0/0

I understood that the above error appears when Nginx doesn't receive responses from upstream on time, and it is marked as unavailable temporarily. I used to get this error before even with 4 upstream servers, but after adding the following additional configuration, it had got resolved:

user nginx;
worker_processes 4;
worker_rlimit_nofile 65535;

load_module "/usr/lib64/nginx/modules/ngx_stream_module.so";

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 10240;
}

stream {
upstream dns_upstreams {
server xxx.xxx.xxx.0:53 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.0:6363 max_fails=2000 fail_timeout=0s;
server xxx.xxx.xxx.187:53 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.187:6363 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.183:53 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.183:6363 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.212:53 max_fails=2000 fail_timeout=30s;
server xxx.xxx.xxx.212:6363 max_fails=2000 fail_timeout=30s;
}

server {
listen 53 udp;
proxy_pass dns_upstreams;
proxy_timeout 1s;
proxy_responses 1;
}
}

Even though this configuration works fine with 4 upstream servers, it doesn't help when I increase the number of servers.

The Nginx server has enough memory and CPU capacity remaining when running with 4 upstream servers as well as 6 upstream servers. And the dnsperf client is not a bottleneck here because it can send much more load in a different setup. Also, the individual upstream server can serve a bit more than 5000 request per second.

I am trying to get some hints about why I am observing more upstream failures and eventual unavailability when I add more servers. If anybody has faced a similar issue in the past and can give me some pointers to solve it, that would of great help.

Thanks,
Ajmal
Subject Author Posted

UDP Load balancer does not scale

ajmalahd May 16, 2017 03:10AM

Re: UDP Load balancer does not scale

Maxim Konovalov May 22, 2017 02:44AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 61
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready