Hello Guys, i recently moved to a server with nginx to handle the load
of my website better. I run a large torrent website which gets about
800.000 unique visitors per day. The machine I use is a Dual Quad
Core, 8GB, Dual SCSI 15k RPM drives in RAID setup. Running LINUX
CENTOS 5 64bit.

My machine always has about 3 to 4GB of its memory free when I check
'top'. The server load never really goes above 4, and usually its
around 1.

*My Problem*
When my site gets really busy, I often experience timeouts. It can
take up to 10 seconds sometimes before I get any response from my
server. Also the bandwidth graph starts to act a little funny.

Bandwidth Graph Example: http://i29.tinypic.com/rtgrk9.jpg

As you can see, normally the BLUE line is a bit higher than the GREEN
one. Which is perfect, but when it gets really busy, it seems the BLUE
line cant keep up with the GREEN one, and it dips. When this happens,
the timeouts start.

Ive tried restarting mysql/nginx/php-fpm and the linux network. These
things do not fix the problem. Also the server load is still low, and
also the parse time is really fast (0.005 per page).

I run on a 100mbit line, and a 100mbit network card. Might it be that
i'm using all of linux's network resources? If so, how could I check
this?

I have included links of my current config files. Im pretty sure mysql
is not the bad guy here, as the parse times are always low. I think
either its a network issue, or nginx has reached some kind of limit,
which must be set higher.

nginx.conf = http://pastebin.com/f65cd3eff
php-fpm.conf = http://pastebin.com/feab061b

I would love to hear some tips/ideas :) thank you very much in
advance.

I believe "apache-like" is not actually usable in php-fpm.conf.

Use "static"

Then tweak your engine - usually roughly 1 child per concurrent php
request you plan on serving up. A php engine can only process one
request at a time, but if your pages are snappy, you can fit multiple
requests in a single engine per second in theory.


On Thu, Jul 9, 2009 at 6:26 AM, Godius<godius@gmail.com> wrote:
>
> Hello Guys, i recently moved to a server with nginx to handle the load
> of my website better. I run a large torrent website which gets about
> 800.000 unique visitors per day. The machine I use is a Dual Quad
> Core, 8GB, Dual SCSI 15k RPM drives in RAID setup. Running LINUX
> CENTOS 5 64bit.
>
> My machine always has about 3 to 4GB of its memory free when I check
> 'top'. The server load never really goes above 4, and usually its
> around 1.
>
> *My Problem*
> When my site gets really busy, I often experience timeouts. It can
> take up to 10 seconds sometimes before I get any response from my
> server. Also the bandwidth graph starts to act a little funny.
>
> Bandwidth Graph Example: http://i29.tinypic.com/rtgrk9.jpg
>
> As you can see, normally the BLUE line is a bit higher than the GREEN
> one. Which is perfect, but when it gets really busy, it seems the BLUE
> line cant keep up with the GREEN one, and it dips. When this happens,
> the timeouts start.
>
> Ive tried restarting mysql/nginx/php-fpm and the linux network. These
> things do not fix the problem. Also the server load is still low, and
> also the parse time is really fast (0.005 per page).
>
> I run on a 100mbit line, and a 100mbit network card. Might it be that
> i'm using all of linux's network resources? If so, how could I check
> this?
>
> I have included links of my current config files. Im pretty sure mysql
> is not the bad guy here, as the parse times are always low. I think
> either its a network issue, or nginx has reached some kind of limit,
> which must be set higher.
>
> nginx.conf = http://pastebin.com/f65cd3eff
> php-fpm.conf = http://pastebin.com/feab061b
>
> I would love to hear some tips/ideas :) thank you very much in
> advance.
>

I'd also change these

<value name="max_requests">1000</value>

to

<value name="max_requests">250</value> or 500

and
<value name="max_children">100</value>

Settings group for 'apache-like' pm style
<value name="apache_like">

Sets the number of server
processes created on startup.
Used only when 'apache-like'
pm_style is selected
<value name="StartServers">100</value>

I usually set the same values for max_children and StartServers, even
though apache isn't used [yet]


On Thu, Jul 9, 2009 at 6:26 AM, Godius<godius@gmail.com> wrote:
>
> Hello Guys, i recently moved to a server with nginx to handle the load
> of my website better. I run a large torrent website which gets about
> 800.000 unique visitors per day. The machine I use is a Dual Quad
> Core, 8GB, Dual SCSI 15k RPM drives in RAID setup. Running LINUX
> CENTOS 5 64bit.
>
> My machine always has about 3 to 4GB of its memory free when I check
> 'top'. The server load never really goes above 4, and usually its
> around 1.
>
> *My Problem*
> When my site gets really busy, I often experience timeouts. It can
> take up to 10 seconds sometimes before I get any response from my
> server. Also the bandwidth graph starts to act a little funny.
>
> Bandwidth Graph Example: http://i29.tinypic.com/rtgrk9.jpg
>
> As you can see, normally the BLUE line is a bit higher than the GREEN
> one. Which is perfect, but when it gets really busy, it seems the BLUE
> line cant keep up with the GREEN one, and it dips. When this happens,
> the timeouts start.
>
> Ive tried restarting mysql/nginx/php-fpm and the linux network. These
> things do not fix the problem. Also the server load is still low, and
> also the parse time is really fast (0.005 per page).
>
> I run on a 100mbit line, and a 100mbit network card. Might it be that
> i'm using all of linux's network resources? If so, how could I check
> this?
>
> I have included links of my current config files. Im pretty sure mysql
> is not the bad guy here, as the parse times are always low. I think
> either its a network issue, or nginx has reached some kind of limit,
> which must be set higher.
>
> nginx.conf = http://pastebin.com/f65cd3eff
> php-fpm.conf = http://pastebin.com/feab061b
>
> I would love to hear some tips/ideas :) thank you very much in
> advance.
>

Maybe I'm misunderstanding something, but the top of that graph is marked as
"12". It says "bits per second" but it's very possible that's mislabeled
and it's Bytes per second, in which case, 12MBps == 12 * 8 == 96mbps. If
so, I'm surprised you're even able to hit that level - that's maxing out not
only the card but the max that you can possibly sustain on an http tcp/ip
connection due to protocol overhead.
Is there an easy way for you to verify the bit vs. byte question?

I'd also do some testing - we found that, at least for us, one php-fpm
process per core (at least on the machines we were using) seemed to be
optimal, so if this is a matter of maxing out the config rather than maxing
out the network card, playing with that might help. We used a hardcoded
number of processes, rather than let it spawn them, and found that worked
pretty well for us. (But we're also still in the testing phase, we haven't
done a large scale roll-out yet.)

Have you tried stracing any of the processes to see what was going on and
what was timing out?

Nicholas


On Thu, Jul 9, 2009 at 9:26 AM, Godius wrote:

>
> Hello Guys, i recently moved to a server with nginx to handle the load
> of my website better. I run a large torrent website which gets about
> 800.000 unique visitors per day. The machine I use is a Dual Quad
> Core, 8GB, Dual SCSI 15k RPM drives in RAID setup. Running LINUX
> CENTOS 5 64bit.
>
> My machine always has about 3 to 4GB of its memory free when I check
> 'top'. The server load never really goes above 4, and usually its
> around 1.
>
> *My Problem*
> When my site gets really busy, I often experience timeouts. It can
> take up to 10 seconds sometimes before I get any response from my
> server. Also the bandwidth graph starts to act a little funny.
>
> Bandwidth Graph Example: http://i29.tinypic.com/rtgrk9.jpg
>
> As you can see, normally the BLUE line is a bit higher than the GREEN
> one. Which is perfect, but when it gets really busy, it seems the BLUE
> line cant keep up with the GREEN one, and it dips. When this happens,
> the timeouts start.
>
> Ive tried restarting mysql/nginx/php-fpm and the linux network. These
> things do not fix the problem. Also the server load is still low, and
> also the parse time is really fast (0.005 per page).
>
> I run on a 100mbit line, and a 100mbit network card. Might it be that
> i'm using all of linux's network resources? If so, how could I check
> this?
>
> I have included links of my current config files. Im pretty sure mysql
> is not the bad guy here, as the parse times are always low. I think
> either its a network issue, or nginx has reached some kind of limit,
> which must be set higher.
>
> nginx.conf = http://pastebin.com/f65cd3eff
> php-fpm.conf = http://pastebin.com/feab061b
>
> I would love to hear some tips/ideas :) thank you very much in
> advance.
>