Welcome! Log In Create A New Profile

Advanced

All workers in 'D' state using sendfile

Drew Wareham
May 12, 2012 06:30AM
Hello,

I have tried to summarize this as much as possible but it's still a lot of
text. I apologize but wanted to make sure that I provide enough
information to explain the issue properly.

I'm hoping that somebody that uses nginx as a high traffic/concurrency
download server will be able to shed some light on this issue. I've tried
as many things as I can think of and everything keeps pointing to it being
an issue with nginx, not the server - but I am of course more than willing
to try any suggestions provided.

*Background:*
Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
Files vary in size from 5MB to 2GB,
All downloads; only very small dynamic content scripts run on these servers
and none take more than 1-3 seconds,
File are hosted on direct-attached AoE storage with a dedicated 10GE link,
Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
(2.6.18-308.4.1.el5.centos.plus).
Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
over 10GE
Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
but obviously not when I'm testing ;)

*Issue:*
After running for a short, but random period of time (5min ~ 90min) all
nginx workers will eventually end up in a 'D' state according to ps/top.
This causes all downloads to run extremely slowly (~25kb/s) but it doesn't
seem to be caused by I/O because an scp of the same file will complete at
the expected speed of ~750MB+/s.

I usually run with worker_processes set to 13, but I've had to raise this
to 50 to prevent the issue. This works short term, but I'm guessing
eventually I will need to restart nginx to fix it.

*Config:*
I'm using sendfile with epoll, and using the following events / http
settings (I've removed the location block with the fastcgi handler, etc):

events {
worker_connections 16384;
use epoll;
}

http {
....

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 0;

....

location /internalAccess/ {
internal;
alias /data/;
}
}



Kind Regards,

Drew
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

All workers in 'D' state using sendfile

Drew Wareham May 12, 2012 06:30AM

Re: All workers in 'D' state using sendfile

Maxim Dounin May 12, 2012 07:20AM

Re: All workers in 'D' state using sendfile

Drew Wareham May 25, 2012 01:00AM

Re: All workers in 'D' state using sendfile

Drew Wareham May 28, 2012 03:10AM

Re: All workers in 'D' state using sendfile

姚伟斌 May 28, 2012 06:08AM

Re: All workers in 'D' state using sendfile

fanboy May 28, 2012 10:32AM

Re: All workers in 'D' state using sendfile

Drew Wareham May 29, 2012 12:52AM

Re: All workers in 'D' state using sendfile

Host DL June 08, 2013 09:48PM

Re: All workers in 'D' state using sendfile

Maxim Dounin June 09, 2013 08:14AM

Re: All workers in 'D' state using sendfile

Host DL June 09, 2013 10:56AM

Re: All workers in 'D' state using sendfile

Maxim Dounin June 10, 2013 06:56AM

Re: All workers in 'D' state using sendfile

Host DL June 10, 2013 07:10AM

Re: All workers in 'D' state using sendfile

Valentin V. Bartenev June 10, 2013 07:32AM

Re: All workers in 'D' state using sendfile

Maxim Dounin June 10, 2013 07:58AM

Re: All workers in 'D' state using sendfile

Host DL June 18, 2013 10:24AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 211
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 494 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready