We are deploying nginx in a docker swarm environment as reverse proxy server. Nginx is running as a container in the swarm, and we only have one upstream backend server, which is a jetty server. Nginx and Jetty are running on separate containers, in same swarm stack.
When we perform load test on the system, some strange behaviors of Nginx occur:
At the begining, Nginx and jetty is very efficient and can serve 10000+ requests per second, however, after receiving 20000 requests, suddenly Nginx becomes very slow (with response time added up to 1 second), but I didn't observe any timeout error in the error log. And at this time, the CPU and memory utilization of Nginx and jetty are both very low, but the stack can only serve several hundred of requests per second.
To exclude the possibility of badly behaved jetty server, I have setup another client in the network and instead of going through Nginx, it directly sends requests to jetty server and jetty server can serve the requests very fast while at the same time, the client->Nginx->Jetty flow is still very slow.
What's more interesting is that in the nginx configuration, I use the hostname of jetty server, which will be resolved to a virtual ip automatically by Swarm. And if I have configured the nginx to point to the real ip of jetty server, the problem mentioned above no longer occurs. I would like to know if anyone has any clue what can cause such weird behavior?
My Nginx configuration:
#user www www; ## Default: nobody
worker_processes 8; ## Default: 1
daemon off;
error_log /work/web-bench1/logs/error.log;
#pid /var/log/nginx/log/nginx.pid;
worker_rlimit_nofile 8192;
events {
multi_accept on;
worker_connections 4096; ## Default: 1024
}
http {
include /work/web-bench1/config/mime.types;
include /work/web-bench1/config/proxy.conf;
index index.html index.htm index.php;
keepalive_timeout 20;
default_type application/octet-stream;
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /work/web-bench1/logs/access.log timed_combined;
sendfile on;
tcp_nopush on;
server_names_hash_bucket_size 128; # this seems to be required for some vhosts
server { # simple reverse-proxy
listen 8888;
server_name receiver-bench1;
location /aelladata/ {
proxy_pass http://receiver:8180/;
proxy_http_version 1.1;
}
}
}
And proxy configuration:
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 1m;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;
