On Thu, Oct 06, 2011 at 02:30:41PM -0400, atadmin wrote:
> Hi,
>
> I a preparing a new web environment with high requirements: 100.000
> concurrents connections per second (sometimes). Every server will
> execute a php script through php5-fpm.
> I am testing where are the limits of nginx (without any php) and how to
> setup the machine for optimize it. I will explain my tests and results:
>
> Test:
>
> 10 servers 4 CPUs, 4 Gb ram, 16Gb HD.
> Local Network: 1Gb (Datacenter network)
>
> 1 Server has a debian squeeze with basic installation (from netinstall
> iso) and nginx from debian repositories (0.7.67-3)
>
> I changed only 2 options for nginx config (i tested with others):
>
> worker_processes 4;
> worker_connections 10240;
>
> I add this lines to /etc/security/limits.conf (restart nginx)
>
> www-data soft nproc 100000
> www-data soft nofile 100000
>
> and for discard I/O issues i mounted /var/log/nginx in ram:
>
> mount -t tmpfs -o nodev,nosuid,noexec,nodiratime,size=2500M none
> /var/log/nginx/
>
> Created static file:
> echo "HOLA">/var/www/a.txt
>
> From the rest of 9 servers with the same basic installation i installed
> apache2-utils and changed: ulimit -n 100000. After just try this
> command:
>
> ab -n 500000 -c 200 http://192.168.1.11/a.txt
>
>
> Really i tested with few server and more with a lot of diferents values
> for ab tool, but i can not get better results:
>
> # awk '{ print $4 }' /var/log/nginx/localhost.access.log |awk -F: '{
> print $2 ":" $3 ":" $4 }'|sort|uniq -c
> [...]
> 22345 19:57:58
> 21088 19:57:59
> 19010 19:58:00
> 20211 19:58:01
> 22469 19:58:02
> 23121 19:58:03
> 22682 19:58:04
> 23105 19:58:05
> 24537 19:58:06
> 22313 19:58:07
> 22406 19:58:08
> 22804 19:58:09
> 23823 19:58:10
> 22280 19:58:11
> 24634 19:58:12
> 22722 19:58:13
> 22429 19:58:14
> 24271 19:58:15
> 20265 19:58:16
> 20678 19:58:17
> 23136 19:58:18
> 22203 19:58:19
> 22521 19:58:20
> 24254 19:58:21
> 23216 19:58:22
> 22587 19:58:23
> 18365 19:58:24
> 22221 19:58:25
> 22123 19:58:26
> 24464 19:58:27
> [...]
>
> Also i tried changing a lot of things in /etc/sysctl.conf (sysctl -p and
> restart nginx) but i didn't see better results.
>
> For example:
>
> net.ipv4.tcp_keepalive_time = 300
> # Avoid a smurf attack
> net.ipv4.icmp_echo_ignore_broadcasts = 1
>
> # Turn on protection for bad icmp error messages
> net.ipv4.icmp_ignore_bogus_error_responses = 1
>
> # Turn on syncookies for SYN flood attack protection
> net.ipv4.tcp_syncookies = 0
>
> # Turn on and log spoofed, source routed, and redirect packets
> net.ipv4.conf.all.log_martians = 1
> net.ipv4.conf.default.log_martians = 1
>
> # No source routed packets here
> net.ipv4.conf.all.accept_source_route = 0
> net.ipv4.conf.default.accept_source_route = 0
>
> # Turn on reverse path filtering
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.conf.default.rp_filter = 1
>
> # Make sure no one can alter the routing tables
> net.ipv4.conf.all.accept_redirects = 0
> net.ipv4.conf.default.accept_redirects = 0
> net.ipv4.conf.all.secure_redirects = 0
> net.ipv4.conf.default.secure_redirects = 0
>
> # Don't act as a router
> net.ipv4.ip_forward = 1
> net.ipv4.conf.all.send_redirects = 0
> net.ipv4.conf.default.send_redirects = 0
>
> # Turn on execshild
> kernel.exec-shield = 1
> kernel.randomize_va_space = 1
>
> # Tuen IPv6
> net.ipv6.conf.default.router_solicitations = 0
> net.ipv6.conf.default.accept_ra_rtr_pref = 0
> net.ipv6.conf.default.accept_ra_pinfo = 0
> net.ipv6.conf.default.accept_ra_defrtr = 0
> net.ipv6.conf.default.autoconf = 0
> net.ipv6.conf.default.dad_transmits = 0
> net.ipv6.conf.default.max_addresses = 1
>
> # Optimization for port usefor LBs
> # Increase system file descriptor limit
> fs.file-max = 655350
>
> # Allow for more PIDs (to reduce rollover problems); may break some
> programs 32768
> kernel.pid_max = 65536
>
> # Increase system IP port limits
> net.ipv4.ip_local_port_range = 1500 65000
>
> # Increase TCP max buffer size setable using setsockopt()
> net.ipv4.tcp_rmem = 4096 87380 33554432
> net.ipv4.tcp_wmem = 4096 65536 33554432
>
> # Increase Linux auto tuning TCP buffer limits
> # min, default, and max number of bytes to use
> # set max to at least 4MB, or higher if you use very high BDP paths
> # Tcp Windows etc
> net.core.rmem_max = 33554432
> net.core.wmem_max = 33554432
> net.core.rmem_default=65536
> net.core.wmem_default=65536
> net.core.netdev_max_backlog = 5000
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_sack = 1
> net.ipv4.tcp_no_metrics_save = 1
>
> With last kernels and autoptimize is not necessary change anything about
> tcp buffers (but i think for this requirements yes).
>
> I was monitoring the machine while tests, CPU usage by nginx is around
> 30%, RAM nothing important, and few I/O traffic, Load <0.50.
>
> Could somebody help me for find where is the bottleneck?
>
> Thanks.
>

Could you be bottle-necked in your testing tool "ab"?

Ken

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx