Hi, I'm using 48 cores of cpu with 256 GB memory for loadbalancing,

I decide to use nginx as loadbalancer to handle +/- 4000 TPS

here is my configuration,

/etc/nginx/nginx.conf
user nginx;
worker_processes 48;
worker_rlimit_nofile 100000;

error_log /var/log/nginx/main_error.log warn;
pid /var/run/nginx.pid;


events {
worker_connections 100000;
use epoll;
multi_accept on;
}


http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

log_format upstreamlog '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"'
' to="$upstream_addr" rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time" msec=$msec';


access_log /var/log/nginx/access.log main;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
send_timeout 1000ms;
include /etc/nginx/conf.d/*.conf;
}




/etc/nginx/conf.d/*.conf
upstream xbox4-responder {
server 10.23.40.34:19001 fail_timeout=0s;
server 10.23.40.35:19001 fail_timeout=0s;
server 10.23.40.33:19001 fail_timeout=0s backup;
server 10.23.40.34:19002 fail_timeout=0s;
server 10.23.40.35:19002 fail_timeout=0s;
server 10.23.40.33:19002 fail_timeout=0s backup;
server 10.23.40.34:19003 fail_timeout=0s;
server 10.23.40.35:19003 fail_timeout=0s;
server 10.23.40.33:19003 fail_timeout=0s backup;
server 10.23.40.34:19004 fail_timeout=0s;
server 10.23.40.35:19004 fail_timeout=0s;
server 10.23.40.33:19004 fail_timeout=0s backup;
server 10.23.40.34:19005 fail_timeout=0s;
server 10.23.40.35:19005 fail_timeout=0s;
server 10.23.40.33:19005 fail_timeout=0s backup;
server 10.23.40.34:19006 fail_timeout=0s;
server 10.23.40.35:19006 fail_timeout=0s;
server 10.23.40.33:19006 fail_timeout=0s backup;
server 10.23.40.34:19007 fail_timeout=0s;
server 10.23.40.35:19007 fail_timeout=0s;
server 10.23.40.33:19007 fail_timeout=0s backup;
server 10.23.40.34:19008 fail_timeout=0s;
server 10.23.40.35:19008 fail_timeout=0s;
server 10.23.40.33:19008 fail_timeout=0s backup;
}

server {
listen 9092;
access_log /var/log/nginx/xbox4.access.log upstreamlog;
error_log /var/log/nginx/xbox4_error.log warn;

location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_pass http://xbox4-responder;
proxy_next_upstream error timeout http_504 non_idempotent;
proxy_next_upstream_tries 3;
proxy_connect_timeout 50ms;
proxy_read_timeout 250ms;
}
}


Notes:
a. If directly accessing 1 port of upstream, without loadbalancer, I'm able to handle 1200TPS per port, using Jmeter.
And I'm expecting to have bigger TPS when using loadbalancer infront of it.

b. I put the non_idempotent, since my request is a post request, but didn't change any value.
<XBOXRequest><requestName>GetSubscriberMenuId</requestName><Subscriber_Number></Subscriber_Number><Source>UMB</Source><Trans_ID>0030180530173955288b1</Trans_ID><Home_POC>JK0</Home_POC><PRICE_PLAN>380408</PRICE_PLAN><PayCat>PRE-PAID</PayCat><Active_End>20180723</Active_End><Grace_End>20180822</Grace_End><Rembal>10</Rembal><IMSI></IMSI><VLR></VLR><IMEI></IMEI><CellID>89f062ad</CellID><Reqtime></Reqtime><Shortcode>*123#</Shortcode></XBOXRequest>

c. I'm using 3 nodes of upstream, with 8 port each.


Questions :

1. I put the value of fail_timeout=0s, expecting that I don't want to make a delay by using the default value. And I'm afraid, there will be a moment that all port will be unavailable if I set > 0s. Does my understanding is correct?
2. My backup upstream never involved on the retries, How to check that the port status not available at that moment? Does the "backup" flag is working or not? Did I put it correctly?

10.193.26.155 - - [04/Jun/2018:09:33:12 +0700] "POST / HTTP/1.1" 504 183 "-" "XBOX Client" to="10.23.40.34:19001, 10.23.40.34:19008, 10.23.40.35:19002" rt="0.751" uct="0.001, 0.000, 0.000" uht="0.251, 0.250, 0.250" urt="0.251, 0.250, 0.250" msec=1528079592.685
10.196.217.153 - - [04/Jun/2018:09:33:12 +0700] "POST / HTTP/1.1" 504 183 "-" "XBOX Client" to="10.23.40.34:19001, 10.23.40.35:19003, 10.23.40.34:19008" rt="0.750" uct="0.000, 0.000, 0.000" uht="0.250, 0.250, 0.250" urt="0.250, 0.250, 0.250" msec=1528079592.694
10.193.26.168 - - [04/Jun/2018:09:33:22 +0700] "POST / HTTP/1.1" 504 183 "-" "XBOX Client" to="10.23.40.34:19004, 10.23.40.34:19001, 10.23.40.35:19006" rt="0.750" uct="0.000, 0.000, 0.000" uht="0.250, 0.250, 0.250" urt="0.250, 0.250, 0.250" msec=1528079602.269
10.196.217.153 - - [04/Jun/2018:09:33:22 +0700] "POST / HTTP/1.1" 504 183 "-" "XBOX Client" to="10.23.40.34:19001, 10.23.40.34:19003, 10.23.40.34:19007" rt="0.750" uct="0.000, 0.000, 0.000" uht="0.250, 0.250, 0.250" urt="0.250, 0.250, 0.250" msec=1528079602.296

does my configuration is to aggressive?

3. What is send_timeout used for? what I'm thinking is that none of the request will be status code 200 if the rt is bigger than 1000 ms, but I saw it's success.

10.196.217.155 - - [04/Jun/2018:13:17:09 +0700] "GET /nbos_offer_detail?MENU_NAME=MN_XL_CLM_8110531_FREE&OPCODE=CPDELIVER&DELIVERMODE=0&DATETIME=20180604131708&SUBGROUP=11&SMSTYPE=&SHORTNO=&SUBSCRIBERID=1869827388 HTTP/1.1" 200 827 "-" "-" to="10.23.40.35:19002" rt="1.187" uct="0.000" uht="0.060" urt="0.060" msec=1528093029.586
10.196.217.155 - - [04/Jun/2018:13:33:15 +0700] "GET /nbos_offer_detail?MENU_NAME=MN_XL_CLM_8110529_NORM&OPCODE=CPDELIVER&DELIVERMODE=0&DATETIME=20180604133314&SUBGROUP=11&SMSTYPE=&SHORTNO=&SUBSCRIBERID=1063847878 HTTP/1.1" 200 804 "-" "-" to="10.23.40.35:19003" rt="1.061" uct="0.000" uht="0.143" urt="0.143" msec=1528093995.252
10.196.217.155 - - [04/Jun/2018:14:07:34 +0700] "GET /nbos_offer_detail?MENU_NAME=MN_XL_CLM_8110530_NORM&OPCODE=CPDELIVER&DELIVERMODE=0&DATETIME=20180604140733&SUBGROUP=11&SMSTYPE=&SHORTNO=&SUBSCRIBERID=1060114969 HTTP/1.1" 200 800 "-" "-" to="10.23.40.34:19007" rt="1.044" uct="0.000" uht="0.014" urt="0.014" msec=1528096054.613
10.22.233.192 - - [04/Jun/2018:07:07:51 +0700] "POST /xbox HTTP/1.1" 200 806 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" to="10.23.40.35:19003" rt="1.875" uct="0.000" uht="0.004" urt="0.004" msec=1528070871.993
10.22.233.192 - - [04/Jun/2018:07:07:54 +0700] "POST /xbox HTTP/1.1" 200 806 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" to="10.23.40.34:19006" rt="1.835" uct="0.000" uht="0.003" urt="0.003" msec=1528070874.698


Thanks