Hello,

I am Nisa, a new learner of Nginx. Honestly I have more than one questions regarding nginx tunning on web server and loadbalancer, but this post will be too long. I would be very grateful, if anyone can help me solve the issue.

So, why is the connection accepted per second unstable in Nginx Loadbalacer?

I have run one loadbalancer and three webservers installed on Ubuntu Server 16.04. All of them are virtual machine ran on Proxmox and have been tunned (I follow the tutorial from here: https://www.unixteacher.org/blog/tuning-nginx-to-handle-more-than-50-000-requests/). The following is the spesification:
2 x 2 core processor
32 GB RAM
40GB HDD

And the following is nginx.conf in Loadbalancer.

***
# /etc/nginx/nginx.conf

user www-data;
worker_processes 8;
worker_priority -15;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 400000;
timer_resolution 10000ms;

events {
worker_connections 20000;
use epoll;
multi_accept on;
#worker_connections 768;
# multi_accept on;
}

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 120;
keepalive_requests 100000;
client_max_body_size 128M;
client_body_timeout 40;
client_header_timeout 40;
client_body_buffer_size 128k;
client_header_buffer_size 8k;
connection_pool_size 8192;
request_pool_size 16k;
server_names_hash_max_size 2048;
server_names_hash_bucket_size 2048;
types_hash_max_size 4086;
proxy_connect_timeout 3600;
proxy_send_timeout 3600;
proxy_read_timeout 3600;
# types_hash_max_size 4084;
server_tokens off;
resolver 127.0.0.1;
resolver_timeout 3s;
reset_timedout_connection on;
send_timeout 60;

# server_names_hash_bucket_size 64;
# server_name_in_redirect off;

open_file_cache max=400000 inactive=30s;
open_file_cache_valid 60s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# SSL Settings
##

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;

##
# Logging Settings
##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

***

The loadbalacer config is on the following:

***
# /etc/nginx/conf.d/lb.conf

upstream myapp3 {
server 192.168.80.4:65080;
server 192.168.80.5:65080;
server 192.168.80.6:65080;
}

server {
listen 192.168.80.7:65080;
access_log /var/log/nginx/lb-access.log;
error_log /var/log/nginx/lb-error.log;

location / {
proxy_pass http://myapp2;
}
}

***

I have run
$ sudo nginx -t
$ sudo nginx -s reload
and there is no traces of failed nginx. As well as when I run
$ sudo service nginx status
it is said it running well

***
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-05-14 10:17:56 WIB; 1h 54min ago
Docs: man:nginx(8)
Process: 16969 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=2)
Process: 16983 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 16979 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 16986 (nginx)
Tasks: 9
Memory: 91.1M
CPU: 1min 6.554s
CGroup: /system.slice/nginx.service
├─16986 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
├─16987 nginx: worker process
├─16988 nginx: worker process
├─16989 nginx: worker process
├─16990 nginx: worker process
├─16991 nginx: worker process
├─16992 nginx: worker process
├─16993 nginx: worker process
└─16994 nginx: worker process

May 14 10:17:56 LB systemd[1]: Starting A high performance web server and a reverse proxy server...
May 14 10:17:56 LB systemd[1]: Started A high performance web server and a reverse proxy server.
***

However, sometimes I got this message in /var/log/nginx/lb-error.log

# tail -f /var/log/nginx/lb-error.log
2020/05/14 07:19:51 [error] 24977#24977: *598307 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.80.1, server: , request: "POST /KreasiNew/api/update HTTP/1.1", upstream: "http://192.168.80.4:65080/KreasiNew/api/update", host: "<ip public>"

noted: I omitted ip public.

and when I monitored the traffic, it looks like the picture attachment.

So, I'd like you to help me pointed out my wrong doings or do you have any sugestions regarding this issue?

best regards,
Nisa