We are seeing problems with Nginx (mostly)locking up the server when
running high loads of HTTPS traffic.
This scenario we had nginx configured to
bind to eth3 but our ssh sessions on eth0 were frozen during this condition as well.
The system restores itself after a few minutes, (the load generation would
have stopped after a minute or two of lockup, that may be what lets things
recover).
We tested different kernels (4.0.4+, 4.0.0+, 3.17.8+ with local patches,
and stock 3.14.27-100.fc19.x86_64, all with same results), different NICs (Intel 10G, Intel 40G),
and Apache as web server.
Apache can sustain about 10.8Gbps of HTTPS traffic and shows no
instability/lockups. nginx maxes out at 2.2Gbps (until it locks up machine).
Some kernel splats indicated some files writing to the file system
journal were blocked > 180 seconds, but they recover, so it is not
a hard lock. The system should not be doing any heavy disk access
since we have 32GB RAM. Swap shows no usage.
=== Scenario ===
Load testing box has a direct connection to eth3->eth3 over 10Gbps port.
Curl clients using https, keepalive, requesting a 1MB file:
1000 clients @ 0.25 req/sec = 243 req/sec, 2.2Gbps tx, load 8.3
400 clients @ 0.65 req/sec = 260 req/sec, 2.2Gbps tx, load 9.2
=== Environment ===
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz
> free
total used free shared buffers cached
Mem: 32840296 1394884 31445412 0 132792 632068
-/+ buffers/cache: 630024 32210272
Swap: 16457724 0 16457724
> cat /etc/issue
Fedora release 19 (Schrödinger’s Cat)
Kernel \r on an \m (\l)
# uname -a
Linux e5-1630-v3-qc 3.14.27-100.fc19.x86_64 #1 SMP Wed Dec 17 19:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# /usr/local/lanforge/nginx/sbin/nginx -v
nginx version: nginx/1.9.1
We are running small patch to allow nginx to bind to a particular interface.. We
tried with this option disabled, and that causes the same trouble. The exact
source is found below:
https://github.com/greearb/nginx/commits/master
We are compiling nginx with these options:
../configure --prefix=/usr/local/lanforge/nginx/ --with-http_ssl_module --with-ipv6 --without-http_rewrite_module
=== Nginx Config ===
worker_processes auto;
worker_rlimit_nofile 100000;
error_log logs/eth3_error.log;
pid /home/lanforge/vr_conf/nginx_eth3.pid;
events {
use epoll;
worker_connections 8096;
multi_accept on;
}
http {
include /usr/local/lanforge/nginx/conf/mime.types;
default_type application/octet-stream;
access_log off;
sendfile on;
directio 1m;
disable_symlinks on;
gzip off;
tcp_nopush on;
tcp_nodelay on;
open_file_cache max=1000 inactive=10s;
open_file_cache_valid 600s;
open_file_cache_min_uses 2000;
open_file_cache_errors off;
etag off;
server {
listen 1.1.1.1:80 so_keepalive=on bind_dev=eth3;
server_name nginx.local nginx web.local web;
location / {
root /var/www/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 1.1.1.1:443 so_keepalive=on ssl bind_dev=eth3;
server_name nginx.local nginx web.local web;
ssl_certificate /usr/local/lanforge/apache.crt;
ssl_certificate_key /usr/local/lanforge/apache.key;
location / {
root /var/www/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
Any help or suggestions is appreciated.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
