Hi,

I a preparing a new web environment with high requirements: 100.000 concurrents connections per second (sometimes). Every server will execute a php script through php5-fpm.
I am testing where are the limits of nginx (without any php) and how to setup the machine for optimize it. I will explain my tests and results:

Test:

10 servers 4 CPUs, 4 Gb ram, 16Gb HD.
Local Network: 1Gb (Datacenter network)

1 Server has a debian squeeze with basic installation (from netinstall iso) and nginx from debian repositories (0.7.67-3)

I changed only 2 options for nginx config (i tested with others):

worker_processes 4;
worker_connections 10240;

I add this lines to /etc/security/limits.conf (restart nginx)

www-data soft nproc 100000
www-data soft nofile 100000

and for discard I/O issues i mounted /var/log/nginx in ram:

mount -t tmpfs -o nodev,nosuid,noexec,nodiratime,size=2500M none /var/log/nginx/

Created static file:
echo "HOLA">/var/www/a.txt

From the rest of 9 servers with the same basic installation i installed apache2-utils and changed: ulimit -n 100000. After just try this command:

ab -n 500000 -c 200 http://192.168.1.11/a.txt


Really i tested with few server and more with a lot of diferents values for ab tool, but i can not get better results:

# awk '{ print $4 }' /var/log/nginx/localhost.access.log |awk -F: '{ print $2 ":" $3 ":" $4 }'|sort|uniq -c
[...]
22345 19:57:58
21088 19:57:59
19010 19:58:00
20211 19:58:01
22469 19:58:02
23121 19:58:03
22682 19:58:04
23105 19:58:05
24537 19:58:06
22313 19:58:07
22406 19:58:08
22804 19:58:09
23823 19:58:10
22280 19:58:11
24634 19:58:12
22722 19:58:13
22429 19:58:14
24271 19:58:15
20265 19:58:16
20678 19:58:17
23136 19:58:18
22203 19:58:19
22521 19:58:20
24254 19:58:21
23216 19:58:22
22587 19:58:23
18365 19:58:24
22221 19:58:25
22123 19:58:26
24464 19:58:27
[...]

Also i tried changing a lot of things in /etc/sysctl.conf (sysctl -p and restart nginx) but i didn't see better results.

For example:

net.ipv4.tcp_keepalive_time = 300
# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 0

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Make sure no one can alter the routing tables
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Don't act as a router
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Turn on execshild
kernel.exec-shield = 1
kernel.randomize_va_space = 1

# Tuen IPv6
net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 0
net.ipv6.conf.default.max_addresses = 1

# Optimization for port usefor LBs
# Increase system file descriptor limit
fs.file-max = 655350

# Allow for more PIDs (to reduce rollover problems); may break some programs 32768
kernel.pid_max = 65536

# Increase system IP port limits
net.ipv4.ip_local_port_range = 1500 65000

# Increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

# Increase Linux auto tuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
# Tcp Windows etc
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default=65536
net.core.wmem_default=65536
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1

With last kernels and autoptimize is not necessary change anything about tcp buffers (but i think for this requirements yes).

I was monitoring the machine while tests, CPU usage by nginx is around 30%, RAM nothing important, and few I/O traffic, Load <0.50.

Could somebody help me for find where is the bottleneck?

Thanks.