January 15, 2012 02:40PM

We use nginx in a latency sensitive application on our own hardware. We have a monitoring system running on the local network that pings the nginx_status page from the sub_status module once per minute. In addition to collecting the stats from the page, it also measures the round trip latency. Latency is often sub 1ms, but sometimes spikes to over 30ms or more and can stay there for up to 5 consecutive minutes. I am trying to identify the source of the latency jitter, if possible.

We run ubuntu 10.04, which has a 100Hz system timer, so I expect that consistently getting below 10ms response is unlikely. Spiking to 30ms, however, feels like there is a config issue somewhere - whether system or nginx. Nginx is a front end for us; all requests are passed to a FCGI backend via unix sockets (no static file serving).

This is an 8 physical core box with hyperthreading turned on. It isn't swapping, has very low disk activity, no TCP/IP issues that I can tell. CPU use ranges from 20 - 40% across all cores (the application and nginx are co-resident). The box handles between 300 and 1500 requests/sec. I have not noticed any correlation between higher overall load and higher ping latency; it seems to be pretty random.

Any suggestions on other things to investigate are appreciated.

Config details follow.



We are running Nginx 1.0.5

nginx.conf (excerpt):
worker_processes 8;
worker_rlimit_nofile 16384;
worker_priority 0;

events {
multi_accept off;
accept_mutex off;
worker_connections 8192;

http {
error_log /var/log/nginx/error.log info buffer=32k;
access_log /var/log/nginx/access.log main buffer=128k;
sendfile off;
tcp_nodelay on;

server {
listen <IP>:80 default_server backlog=1024;

sysctl.conf (excerpt):

vm.vfs_cache_pressure = 50
fs.file-max = 65535
net.ipv4.ip_local_port_range = 2000 65000
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 30
Subject Author Posted

Inconsistent latency to sub_status

dbanks January 15, 2012 02:40PM

Re: Inconsistent latency to sub_status

dbanks April 27, 2012 10:05PM

Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 287
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready