hi guys, i have a problem.

i have a high end server - dual quad core xeons, 16 gb ram, 4 x 120 gb
ssd's in raid 10, and a gigabit pipe. i only host 1 site on it, and
from time to time the php simply locks up, restarts, and during that
time, the users experience 5-10 seconds load times on pages.

i have around 1000 1500 concurrent users at peak times on the website.

here's what happens in the php-fpm.log:

http://pastebin.ca/2131186

some of those scripts that you see as executing slow, are like 4 - 5
lines of code, so it's pretty impossible to be a coding issue.

here is my php-fpm.conf:

http://pastebin.ca/2131189

if you guys couuld help me shed some light on this i would much
appreciate.

I could be wrong, but 500 in pm.max_requests = 500 seems low. We use 10,000.

Also, why in gods name do you need to start 256 workers? I have played around with workers and found usually 2x-3x the number of cores is usually sufficient. Though, if you have a lot of blocking, then yes starting more workers will result in improved performance.

Also, do you really need: catch_workers_output = yes

On Thursday, March 22, 2012 at 12:19 PM, katmai wrote:
> coding issue.
>

Also, if you're running Linux, the following parameter is wrong and should be commented, so it takes default value (safe).

; Specify the event mechanism FPM will use. The following is available:
; - select (any POSIX os)
; - poll (any POSIX os)
; - epoll (linux >= 2.5.44)
; - kqueue (FreeBSD >= 4.1, OpenBSD >= 2.9, NetBSD >= 2.0)
; - /dev/poll (Solaris >= 7)
; - port (Solaris >= 10)
; Default Value: not set (auto detection)
events.mechanism = kqueue

If you're still experiencing issues, post your nginx.conf as well.

Andrejs

> I could be wrong, but 500 in pm.max_requests = 500 seems low. We use
> 10,000.

500 is really, really low. At 1500 requests per second you have 3 child
processes killed and respawned every second.
I used to have it set to 500 000, and it was a bit too high - php
started to lock up too (php has memory leaks too..), so the recommended
value would be somewhere in: 10 000 - 50 000 range.

> Also, why in gods name do you need to start 256 workers? I have played
> around with workers and found usually 2x-3x the number of cores is
> usually sufficient. Though, if you have a lot of blocking, then yes
> starting more workers will result in improved performance.

Most likely to bury server or at least make it swap like a hot couple
in the 70's ;-)
Max children is 512. With at least 30MB RAM per child + application
load = 50MB+/child. 50MB * 512 = 25GB RAM used, or more than server
physical memory. Even the minimal 256 children = 13GB

I'd start with:

pm.max_requests = 20000
pm = static
pm.max_children = 64

this would give moderate children respawn rate (1 every 13 seconds in
load peak) and reasonable number of children (8 per physical cpu core)
- I can't really see any advantages of dynamic number of children
except for multiple pool setups. If you have a lot of blocking
increasing number of children will smooth the slowdowns over, but the
general performance will decrease (very slightly, but process switching
isn't free). Experiment with values in 30-130 range and see which is
the best one for you.

BTW I hope you're not using php for image processing (you left the IP
of website frontend in config, so I had a look :P ) - this tends to be
a recipe for disaster under higher loads.

--
Maciej Lisiewski

thanks for the suggestions.

i took your suggestions and i did:

pm.start_servers = 48
pm.min_spare_servers = 32
pm.max_spare_servers = 64
pm.max_requests = 10000

taking it from here.

why are you more comfortable with pm.static instead of dynamic?

no, the front end is separated from the convertor :)

On 3/22/2012 9:33 PM, Maciej Lisiewski wrote:
>> I could be wrong, but 500 in pm.max_requests = 500 seems low. We use
>> 10,000.
>
> 500 is really, really low. At 1500 requests per second you have 3
> child processes killed and respawned every second.
> I used to have it set to 500 000, and it was a bit too high - php
> started to lock up too (php has memory leaks too..), so the
> recommended value would be somewhere in: 10 000 - 50 000 range.
>
>> Also, why in gods name do you need to start 256 workers? I have
>> played around with workers and found usually 2x-3x the number of
>> cores is usually sufficient. Though, if you have a lot of blocking,
>> then yes starting more workers will result in improved performance.
>
> Most likely to bury server or at least make it swap like a hot couple
> in the 70's ;-)
> Max children is 512. With at least 30MB RAM per child + application
> load = 50MB+/child. 50MB * 512 = 25GB RAM used, or more than server
> physical memory. Even the minimal 256 children = 13GB
>
> I'd start with:
>
> pm.max_requests = 20000
> pm = static
> pm.max_children = 64
>
> this would give moderate children respawn rate (1 every 13 seconds in
> load peak) and reasonable number of children (8 per physical cpu core)
> - I can't really see any advantages of dynamic number of children
> except for multiple pool setups. If you have a lot of blocking
> increasing number of children will smooth the slowdowns over, but the
> general performance will decrease (very slightly, but process
> switching isn't free). Experiment with values in 30-130 range and see
> which is the best one for you.
>
> BTW I hope you're not using php for image processing (you left the IP
> of website frontend in config, so I had a look :P ) - this tends to be
> a recipe for disaster under higher loads.
>

Agree unless you need dynamic pools, in order to save memory (useful for shared hosts) might as well just go with static. You don't care if you have more idle workers than necessary really.

On Thursday, March 22, 2012 at 2:03 PM, StefanitaRares Dumitrescu wrote:
> thanks for the suggestions.
>
> i took your suggestions and i did:
>
> pm.start_servers = 48
> pm.min_spare_servers = 32
> pm.max_spare_servers = 64
> pm.max_requests = 10000
>
> taking it from here.
>
> why are you more comfortable with pm.static instead of dynamic?
>
> no, the front end is separated from the convertor :)
>
> On 3/22/2012 9:33 PM, Maciej Lisiewski wrote:
> > > I could be wrong, but 500 in pm.max_requests = 500 seems low. We use
> > > 10,000.
> > >
> >
> >
> > 500 is really, really low. At 1500 requests per second you have 3
> > child processes killed and respawned every second.
> > I used to have it set to 500 000, and it was a bit too high - php
> > started to lock up too (php has memory leaks too..), so the
> > recommended value would be somewhere in: 10 000 - 50 000 range.
> >
> > > Also, why in gods name do you need to start 256 workers? I have
> > > played around with workers and found usually 2x-3x the number of
> > > cores is usually sufficient. Though, if you have a lot of blocking,
> > > then yes starting more workers will result in improved performance.
> > >
> >
> >
> > Most likely to bury server or at least make it swap like a hot couple
> > in the 70's ;-)
> > Max children is 512. With at least 30MB RAM per child + application
> > load = 50MB+/child. 50MB * 512 = 25GB RAM used, or more than server
> > physical memory. Even the minimal 256 children = 13GB
> >
> > I'd start with:
> >
> > pm.max_requests = 20000
> > pm = static
> > pm.max_children = 64
> >
> > this would give moderate children respawn rate (1 every 13 seconds in
> > load peak) and reasonable number of children (8 per physical cpu core)
> > - I can't really see any advantages of dynamic number of children
> > except for multiple pool setups. If you have a lot of blocking
> > increasing number of children will smooth the slowdowns over, but the
> > general performance will decrease (very slightly, but process
> > switching isn't free). Experiment with values in 30-130 range and see
> > which is the best one for you.
> >
> > BTW I hope you're not using php for image processing (you left the IP
> > of website frontend in config, so I had a look :P ) - this tends to be
> > a recipe for disaster under higher loads.
> >
>
>
>

it just happened again, but this time i found something interesting:

http://pastebin.ca/2131217

1.
[22-Mar-201214:30:01.779254]DEBUG: pid65269,
fpm_pctl_perform_idle_server_maintenance(), line428:[pool
www]4child(ren)have been created dynamically
2.
[22-Mar-201214:30:02.781856]DEBUG: pid65269,
fpm_pctl_perform_idle_server_maintenance(), line379:[pool
www]currently71active children,0spare children,71running children.
Spawning rate8
3.
[22-Mar-201214:30:02.781883]WARNING: pid65269,
fpm_pctl_perform_idle_server_maintenance(), line400:[pool www]seems
busy(you may need to increase pm.start_servers, or
pm.min/max_spare_servers), spawning8children, there are0idle,
and71total children
4.


this is the most important section to be more exact. how come do i have
71 active children? each children can handle 10000 connections, so in
theory there's 700 000 requests on the server at that specific time?


On 3/22/2012 10:15 PM, Justin Keller wrote:
> Agree unless you need dynamic pools, in order to save memory (useful
> for shared hosts) might as well just go with static. You don't care if
> you have more idle workers than necessary really.
>
> On Thursday, March 22, 2012 at 2:03 PM, StefanitaRares Dumitrescu wrote:
>> thanks for the suggestions.
>>
>> i took your suggestions and i did:
>>
>> pm.start_servers = 48
>> pm.min_spare_servers = 32
>> pm.max_spare_servers = 64
>> pm.max_requests = 10000
>>
>> taking it from here.
>>
>> why are you more comfortable with pm.static instead of dynamic?
>>
>> no, the front end is separated from the convertor :)
>>
>> On 3/22/2012 9:33 PM, Maciej Lisiewski wrote:
>>>> I could be wrong, but 500 in pm.max_requests = 500 seems low. We use
>>>> 10,000.
>>>
>>> 500 is really, really low. At 1500 requests per second you have 3
>>> child processes killed and respawned every second.
>>> I used to have it set to 500 000, and it was a bit too high - php
>>> started to lock up too (php has memory leaks too..), so the
>>> recommended value would be somewhere in: 10 000 - 50 000 range.
>>>
>>>> Also, why in gods name do you need to start 256 workers? I have
>>>> played around with workers and found usually 2x-3x the number of
>>>> cores is usually sufficient. Though, if you have a lot of blocking,
>>>> then yes starting more workers will result in improved performance.
>>>
>>> Most likely to bury server or at least make it swap like a hot couple
>>> in the 70's ;-)
>>> Max children is 512. With at least 30MB RAM per child + application
>>> load = 50MB+/child. 50MB * 512 = 25GB RAM used, or more than server
>>> physical memory. Even the minimal 256 children = 13GB
>>>
>>> I'd start with:
>>>
>>> pm.max_requests = 20000
>>> pm = static
>>> pm.max_children = 64
>>>
>>> this would give moderate children respawn rate (1 every 13 seconds in
>>> load peak) and reasonable number of children (8 per physical cpu core)
>>> - I can't really see any advantages of dynamic number of children
>>> except for multiple pool setups. If you have a lot of blocking
>>> increasing number of children will smooth the slowdowns over, but the
>>> general performance will decrease (very slightly, but process
>>> switching isn't free). Experiment with values in 30-130 range and see
>>> which is the best one for you.
>>>
>>> BTW I hope you're not using php for image processing (you left the IP
>>> of website frontend in config, so I had a look :P ) - this tends to be
>>> a recipe for disaster under higher loads.
>

> this is the most important section to be more exact. how come do i
> have 71 active children? each children can handle 10000 connections,
> so in theory there's 700 000 requests on the server at that specific
> time?

1 child can handle 1 request at the time. If at any given time there
are more requests than children they are put in a queue.
pm.max_requests is the maximum number of requests a child process will
process, one after another, before being automatically killed.

The log section you have quoted means: there are no idle child
processes, I would spawn more, but I have hit the limit set in
configuration file, so I'm going to queue requests instead. It's not an
error, it's the expected behavior. In addition to that there is nothing
in the log you put on pastebin that indicates error of any kind - no
unresponsive child processes, no timeout notices - you had that with
your previous configuration.

So, what exactly has happened again? Please describe what you have
observed, because there is nothing in logs.


As for static vs dynamic:
Dynamic is useful if you have multiple php process pools and load
varies for each of them. If load for one of the pools spikes the number
of children is automatically increased and when the spike ends it
returns to normal.
Static is prefered if you have a single php process pool handling one
(like in your case) php app - since the only task of the server is run
this one app there is no reason for it not to run at full capacity at
all times without costly and slow process spawning.


--
Maciej Lisiewski

any opinions/insight on dynamic, versus on demand ?

when do you recommend one use on demand versus dynamic?



On Thursday, March 22, 2012 at 5:23 PM, Maciej Lisiewski wrote:

> > this is the most important section to be more exact. how come do i
> > have 71 active children? each children can handle 10000 connections,
> > so in theory there's 700 000 requests on the server at that specific
> > time?
> >
>
>
> 1 child can handle 1 request at the time. If at any given time there
> are more requests than children they are put in a queue.
> pm.max_requests is the maximum number of requests a child process will
> process, one after another, before being automatically killed.
>
> The log section you have quoted means: there are no idle child
> processes, I would spawn more, but I have hit the limit set in
> configuration file, so I'm going to queue requests instead. It's not an
> error, it's the expected behavior. In addition to that there is nothing
> in the log you put on pastebin that indicates error of any kind - no
> unresponsive child processes, no timeout notices - you had that with
> your previous configuration.
>
> So, what exactly has happened again? Please describe what you have
> observed, because there is nothing in logs.
>
>
> As for static vs dynamic:
> Dynamic is useful if you have multiple php process pools and load
> varies for each of them. If load for one of the pools spikes the number
> of children is automatically increased and when the spike ends it
> returns to normal.
> Static is prefered if you have a single php process pool handling one
> (like in your case) php app - since the only task of the server is run
> this one app there is no reason for it not to run at full capacity at
> all times without costly and slow process spawning.
>
>
> --
> Maciej Lisiewski
>
>

the php lock happened again, so i set the pm to static instead of
dynamic, but i got the same thing:

http://pastebin.ca/2131354

the pid is the php-fpm master process:

>ps aux |grep 74158
root 74158 0.0 0.1 172364 20352 ?? Ss 10:24PM 0:00.18 php-fpm:
master process (/usr/local/etc/php-fpm.conf) (php-fpm)

i installed httpry and monitored the traffic but i found nothing
suspicious at the time of the php lockup.

On 3/22/2012 11:23 PM, Maciej Lisiewski wrote:
>> this is the most important section to be more exact. how come do i
>> have 71 active children? each children can handle 10000 connections,
>> so in theory there's 700 000 requests on the server at that specific
>> time?
>
> 1 child can handle 1 request at the time. If at any given time there
> are more requests than children they are put in a queue.
> pm.max_requests is the maximum number of requests a child process will
> process, one after another, before being automatically killed.
>
> The log section you have quoted means: there are no idle child
> processes, I would spawn more, but I have hit the limit set in
> configuration file, so I'm going to queue requests instead. It's not
> an error, it's the expected behavior. In addition to that there is
> nothing in the log you put on pastebin that indicates error of any
> kind - no unresponsive child processes, no timeout notices - you had
> that with your previous configuration.
>
> So, what exactly has happened again? Please describe what you have
> observed, because there is nothing in logs.
>
>
> As for static vs dynamic:
> Dynamic is useful if you have multiple php process pools and load
> varies for each of them. If load for one of the pools spikes the
> number of children is automatically increased and when the spike ends
> it returns to normal.
> Static is prefered if you have a single php process pool handling one
> (like in your case) php app - since the only task of the server is run
> this one app there is no reason for it not to run at full capacity at
> all times without costly and slow process spawning.
>
>

> the php lock happened again, so i set the pm to static instead of
> dynamic, but i got the same thing:

Quick question:
Is you max_execution_time (in php.ini, not fpm config) higher than 10?
If it is can you set it to a value in 5-9 range and test again?

--
Maciej Lisiewski

i narrowed it down eventually.

i have some sql tables that got about 1 2 mil each. we got proper
indexes in place. i had some cron jobs that do periodic updates to the
database, locking those tables. i was able to reproduce the lock up at
the exact time when those crons run, so we are now optimizing.
gotta try to find a proper solution so that data is being updated but
the db doesn't go bananas.

On 3/23/2012 11:06 AM, Maciej Lisiewski wrote:
>> the php lock happened again, so i set the pm to static instead of
>> dynamic, but i got the same thing:
>
> Quick question:
> Is you max_execution_time (in php.ini, not fpm config) higher than 10?
> If it is can you set it to a value in 5-9 range and test again?
>

So why don't you start cron job from a shell? In other words, I don't see a point locking a www pool with time-cost task. Run your cron job from console, omit http://.

I think there's a problem, not in locking a db.

-----Original Message-----
From: highload-php-en@googlegroups.com [mailto:highload-php-en@googlegroups.com] On Behalf Of StefanitaRares Dumitrescu
Sent: Friday, March 23, 2012 12:05 PM
To: highload-php-en@googlegroups.com
Subject: Re: php-fpm 5.3.10 / nginx - php locks up

i narrowed it down eventually.

i have some sql tables that got about 1 2 mil each. we got proper indexes in place. i had some cron jobs that do periodic updates to the database, locking those tables. i was able to reproduce the lock up at the exact time when those crons run, so we are now optimizing.
gotta try to find a proper solution so that data is being updated but the db doesn't go bananas.

On 3/23/2012 11:06 AM, Maciej Lisiewski wrote:
>> the php lock happened again, so i set the pm to static instead of
>> dynamic, but i got the same thing:
>
> Quick question:
> Is you max_execution_time (in php.ini, not fpm config) higher than 10?
> If it is can you set it to a value in 5-9 range and test again?
>

On 03/23/2012 12:05 PM, StefanitaRares Dumitrescu wrote:
> i narrowed it down eventually.
>
> i have some sql tables that got about 1 2 mil each. we got proper indexes
> in place. i had some cron jobs that do periodic updates to the database,
> locking those tables. i was able to reproduce the lock up at the exact time
> when those crons run, so we are now optimizing.
> gotta try to find a proper solution so that data is being updated but the
> db doesn't go bananas.

Stop using MyISAM tables and switch to InnoDB. That way you no longer have
the table locking issue.

Regards,
Dennis

> i narrowed it down eventually.
>
> i have some sql tables that got about 1 2 mil each. we got proper
> indexes in place. i had some cron jobs that do periodic updates to
> the database, locking those tables. i was able to reproduce the lock
> up at the exact time when those crons run, so we are now optimizing.
> gotta try to find a proper solution so that data is being updated but
> the db doesn't go bananas.
>
> On 3/23/2012 11:06 AM, Maciej Lisiewski wrote:
>>> the php lock happened again, so i set the pm to static instead of
>>> dynamic, but i got the same thing:
>>
>> Quick question:
>> Is you max_execution_time (in php.ini, not fpm config) higher than 10?
>> If it is can you set it to a value in 5-9 range and test again?
>>
>
>
That's what I started suspecting, hence the max_execution_time question
With all the errors/warnings that could be caused by fpm configuration
eliminated, the next step was to find out if it was caused by php code
being blocked.
Anyway I'm glad you found it. Fixing table locking problems is usually
simple - just switch those tables to innodb - it has row locking instead
of table locking.

--
Maciej Lisiewski

> any opinions/insight on dynamic, versus on demand ?
>
> when do you recommend one use on demand versus dynamic?
>
Static, best for: single pool running single app/group of apps.
Dynamic, best for: multiple pools, all/most of them handling 10k+
requests/day
On demand, best for: large number of pools, many of which handle very
low load (few hundred requests/day or less) - your typical shared
hosting setup with hundreds of low traffic sites per server

--
Maciej Lisiewski

yep that was the next logical step to go check the application.

On 3/23/2012 3:25 PM, Maciej Lisiewski wrote:
>
>> i narrowed it down eventually.
>>
>> i have some sql tables that got about 1 2 mil each. we got proper
>> indexes in place. i had some cron jobs that do periodic updates to
>> the database, locking those tables. i was able to reproduce the lock
>> up at the exact time when those crons run, so we are now optimizing.
>> gotta try to find a proper solution so that data is being updated but
>> the db doesn't go bananas.
>>
>> On 3/23/2012 11:06 AM, Maciej Lisiewski wrote:
>>>> the php lock happened again, so i set the pm to static instead of
>>>> dynamic, but i got the same thing:
>>>
>>> Quick question:
>>> Is you max_execution_time (in php.ini, not fpm config) higher than 10?
>>> If it is can you set it to a value in 5-9 range and test again?
>>>
>>
>>
> That's what I started suspecting, hence the max_execution_time question
> With all the errors/warnings that could be caused by fpm configuration
> eliminated, the next step was to find out if it was caused by php code
> being blocked.
> Anyway I'm glad you found it. Fixing table locking problems is usually
> simple - just switch those tables to innodb - it has row locking
> instead of table locking.
>

Thanks a lot Maciej for sharing all this information and insights.
Great resource for sure.

On Fri, Mar 23, 2012 at 11:34 AM, Maciej Lisiewski
<maciej.lisiewski@gmail.com> wrote:
> Static, best for: single pool running single app/group of apps.
> Dynamic, best for: multiple pools, all/most of them handling 10k+
> requests/day
> On demand, best for: large number of pools, many of which handle very low
> load (few hundred requests/day or less) - your typical shared hosting setup
> with hundreds of low traffic sites per server

+1 - thanks Maciej - very helpful

On Saturday, 24 March 2012 02:39:18 UTC+11, juangiordana wrote:
>
> Thanks a lot Maciej for sharing all this information and insights.
> Great resource for sure.
>
> On Fri, Mar 23, 2012 at 11:34 AM, Maciej Lisiewski
> <maciej...> wrote:
> > Static, best for: single pool running single app/group of apps.
> > Dynamic, best for: multiple pools, all/most of them handling 10k+
> > requests/day
> > On demand, best for: large number of pools, many of which handle very low
> > load (few hundred requests/day or less) - your typical shared hosting
> setup
> > with hundreds of low traffic sites per server
>