All files from this thread

File Name	File Size		Posted by	Date
Screen shot 2011-06-27 at 10.08.08 AM (2).png	321.6 KB	open | download	Paras pradhan	06/27/2011	Read message

Hi,

I am running horde web mail under Apache 2.2.3 , php5.3 and php-fpm
5.3.6. After few hours of restarting apache all my server cores are
100% busy and the load is very high around 20.

sample o/p

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
11639 apache 25 0 302m 40m 6392 R 99.9 0.1 10873:37 php-
fpm
15137 apache 25 0 296m 36m 5508 R 99.9 0.1 295:05.82 php-
fpm
16717 apache 25 0 297m 37m 5612 R 99.9 0.1 11351:56 php-
fpm
19359 apache 25 0 289m 30m 5896 R 99.9 0.0 162:21.60 php-
fpm
23122 apache 25 0 296m 31m 5832 R 99.9 0.0 10868:44 php-
fpm


Anything I can do reduce the cpu usage?

Thanks
Paras.

Is there anything in your Access / Error logs? I'm personally running the
latest software versions and never seen the load quite that high. Almost
looks like a loop perhaps... *shrug*

On Fri, Jun 24, 2011 at 3:22 PM, Paras <pradhanparas@gmail.com> wrote:

> Hi,
>
> I am running horde web mail under Apache 2.2.3 , php5.3 and php-fpm
> 5.3.6. After few hours of restarting apache all my server cores are
> 100% busy and the load is very high around 20.
>
> sample o/p
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 11639 apache 25 0 302m 40m 6392 R 99.9 0.1 10873:37 php-
> fpm
> 15137 apache 25 0 296m 36m 5508 R 99.9 0.1 295:05.82 php-
> fpm
> 16717 apache 25 0 297m 37m 5612 R 99.9 0.1 11351:56 php-
> fpm
> 19359 apache 25 0 289m 30m 5896 R 99.9 0.0 162:21.60 php-
> fpm
> 23122 apache 25 0 296m 31m 5832 R 99.9 0.0 10868:44 php-
> fpm
>
>
> Anything I can do reduce the cpu usage?
>
> Thanks
> Paras.
>

Well not seeing anything in access and error logs of apache.

Paras.


On Jun 24, 3:27 pm, "Sammie S. Taunton" <diem...@gmail.com> wrote:
> Is there anything in your Access / Error logs? I'm personally running the
> latest software versions and never seen the load quite that high. Almost
> looks like a loop perhaps... *shrug*
>
>
>
> On Fri, Jun 24, 2011 at 3:22 PM, Paras <pradhanpa...@gmail.com> wrote:
> > Hi,
>
> > I am running horde web mail under Apache 2.2.3 , php5.3 and php-fpm
> > 5.3.6. After few hours of restarting apache all my server cores are
> > 100% busy and the load is very high around 20.
>
> > sample o/p
>
> >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> > COMMAND
> > 11639 apache    25   0  302m  40m 6392 R 99.9  0.1  10873:37 php-
> > fpm
> > 15137 apache    25   0  296m  36m 5508 R 99.9  0.1 295:05.82 php-
> > fpm
> > 16717 apache    25   0  297m  37m 5612 R 99.9  0.1  11351:56 php-
> > fpm
> > 19359 apache    25   0  289m  30m 5896 R 99.9  0.0 162:21.60 php-
> > fpm
> > 23122 apache    25   0  296m  31m 5832 R 99.9  0.0  10868:44 php-
> > fpm
>
> > Anything I can do reduce the cpu usage?
>
> > Thanks
> > Paras.

Here is the screenshot of the situation. I wonder why the php-fpm processess
are living for 200+ hrs when I have pm.max_requests = 500


Any help will be highly appreciated.

Thanks!
Paras.





On Fri, Jun 24, 2011 at 3:27 PM, Sammie S. Taunton <diemuzi@gmail.com>wrote:

> Is there anything in your Access / Error logs? I'm personally running the
> latest software versions and never seen the load quite that high. Almost
> looks like a loop perhaps... *shrug*
>
>
> On Fri, Jun 24, 2011 at 3:22 PM, Paras <pradhanparas@gmail.com> wrote:
>
>> Hi,
>>
>> I am running horde web mail under Apache 2.2.3 , php5.3 and php-fpm
>> 5.3.6. After few hours of restarting apache all my server cores are
>> 100% busy and the load is very high around 20.
>>
>> sample o/p
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 11639 apache 25 0 302m 40m 6392 R 99.9 0.1 10873:37 php-
>> fpm
>> 15137 apache 25 0 296m 36m 5508 R 99.9 0.1 295:05.82 php-
>> fpm
>> 16717 apache 25 0 297m 37m 5612 R 99.9 0.1 11351:56 php-
>> fpm
>> 19359 apache 25 0 289m 30m 5896 R 99.9 0.0 162:21.60 php-
>> fpm
>> 23122 apache 25 0 296m 31m 5832 R 99.9 0.0 10868:44 php-
>> fpm
>>
>>
>> Anything I can do reduce the cpu usage?
>>
>> Thanks
>> Paras.
>>
>
>

On Mon, Jun 27, 2011 at 4:11 PM, Paras pradhan <pradhanparas@gmail.com> wrote:
> Here is the screenshot of the situation. I wonder why the php-fpm processess
> are living for 200+ hrs when I have pm.max_requests = 500
>

Would you happen to be launching forked processes in the background
for instance? I've noticed php-fpm hanging when commands using
system() were hanging for a long time.

Try inspecting the pids with something like pstree and see if anything
else might might be holding the processes back.


--
David Coallier,
CTO Orchestra Platform Ltd, https://orchestra.io
@davidcoallier, skype://david.coallier

I can see this from pstree

├─httpd───61*[httpd]
├─php-fpm───50*[php-fpm]

So I am assuming nothing is holding the processes from dying (?)

Thanks,
Paras.



On Mon, Jun 27, 2011 at 11:03 AM, David Coallier <david@orchestra.io> wrote:

> On Mon, Jun 27, 2011 at 4:11 PM, Paras pradhan <pradhanparas@gmail.com>
> wrote:
> > Here is the screenshot of the situation. I wonder why the php-fpm
> processess
> > are living for 200+ hrs when I have pm.max_requests = 500
> >
>
> Would you happen to be launching forked processes in the background
> for instance? I've noticed php-fpm hanging when commands using
> system() were hanging for a long time.
>
> Try inspecting the pids with something like pstree and see if anything
> else might might be holding the processes back.
>
>
> --
> David Coallier,
> CTO Orchestra Platform Ltd, https://orchestra.io
> @davidcoallier, skype://david.coallier
>

On 06/25/2011 12:22 AM, Paras wrote:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 11639 apache 25 0 302m 40m 6392 R 99.9 0.1 10873:37 php-
> fpm

# gdb attach 11639 (or a PID of any other 'stuck' php-fpm process)
gdb> bt

And paste the output here.

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP

Here it is:

http://pastebin.com/jyqeiJGH


Thanks,
Paras.

On Mon, Jun 27, 2011 at 2:04 PM, Antony Dovgal <tony@daylessday.org> wrote:

> On 06/25/2011 12:22 AM, Paras wrote:
>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 11639 apache 25 0 302m 40m 6392 R 99.9 0.1 10873:37 php-
>> fpm
>>
>
> # gdb attach 11639 (or a PID of any other 'stuck' php-fpm process)
> gdb> bt
>
> And paste the output here.
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime profiling for PHP
>

On 06/27/2011 11:16 PM, Paras pradhan wrote:
> Here it is:
>
> http://pastebin.com/jyqeiJGH

The backtrace is quite useless without debug symbols..
Did you build that binary yourself or did you use some kind of package?
Can you rebuild it with CLFAGS="-g3 -O0" or install the debug package?

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP

That was gdb package from redhat repo. I also built gdb from src using
CLFAGS="-g3
-O0" . And I am still seeing "no debugging symbols" from that too.

Here's the o/p from the new gdb

http://pastebin.com/099B2FNE

Paras.

On Mon, Jun 27, 2011 at 2:26 PM, Antony Dovgal <tony@daylessday.org> wrote:

> On 06/27/2011 11:16 PM, Paras pradhan wrote:
>
>> Here it is:
>>
>> http://pastebin.com/jyqeiJGH
>>
>
> The backtrace is quite useless without debug symbols..
> Did you build that binary yourself or did you use some kind of package?
> Can you rebuild it with CLFAGS="-g3 -O0" or install the debug package?
>
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime profiling for PHP
>

Antony refers to PHP. (You must build PHP with debugging symbols if you
want to debug PHP).

On 06/27/2011 04:54 PM, Paras pradhan wrote:
> That was gdb package from redhat repo. I also built gdb from src using
> CLFAGS="-g3 -O0" . And I am still seeing "no debugging symbols" from
> that too.
>
> Here's the o/p from the new gdb
>
> http://pastebin.com/099B2FNE
>
> Paras.
>
> On Mon, Jun 27, 2011 at 2:26 PM, Antony Dovgal <tony@daylessday.org
> <mailto:tony@daylessday.org>> wrote:
>
> On 06/27/2011 11:16 PM, Paras pradhan wrote:
>
> Here it is:
>
> http://pastebin.com/jyqeiJGH
>
>
> The backtrace is quite useless without debug symbols..
> Did you build that binary yourself or did you use some kind of package?
> Can you rebuild it with CLFAGS="-g3 -O0" or install the debug package?
>
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime profiling for PHP
>
>

On 06/27/2011 11:54 PM, Paras pradhan wrote:
> That was gdb package from redhat repo. I also built gdb from src using CLFAGS="-g3 -O0" . And I am still seeing "no debugging symbols" from that too.

Yes, I was talking about PHP binary, of course.

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP

Ok I think i got it this time.. here it is

http://pastebin.com/5qCbhgzZ

Paras.


On Mon, Jun 27, 2011 at 3:05 PM, Antony Dovgal <tony@daylessday.org> wrote:

> On 06/27/2011 11:54 PM, Paras pradhan wrote:
>
>> That was gdb package from redhat repo. I also built gdb from src using
>> CLFAGS="-g3 -O0" . And I am still seeing "no debugging symbols" from that
>> too.
>>
>
> Yes, I was talking about PHP binary, of course.
>
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime profiling for PHP
>

On 06/28/2011 12:11 AM, Paras pradhan wrote:
> Ok I think i got it this time.. here it is
>
> http://pastebin.com/5qCbhgzZ

Looks quite alright to me.
It's just executing a script, calling some funcs and reading some array dimensions using ArrayAccess.
Are you sure there's anything wrong with this particular process?

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP

Well perfomance wise I am not having issues at this time but don't know when
I will be having 5 times more users in the future. My concern right now is
why all of my CPU cores are 100% busy and load is around 20.

Thanks
Paras.

On Mon, Jun 27, 2011 at 3:35 PM, Antony Dovgal <tony@daylessday.org> wrote:

> On 06/28/2011 12:11 AM, Paras pradhan wrote:
>
>> Ok I think i got it this time.. here it is
>>
>> http://pastebin.com/5qCbhgzZ
>>
>
> Looks quite alright to me.
> It's just executing a script, calling some funcs and reading some array
> dimensions using ArrayAccess.
> Are you sure there's anything wrong with this particular process?
>
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime profiling for PHP
>

2011/6/27 Paras pradhan <pradhanparas@gmail.com>:
> Well perfomance wise I am not having issues at this time but don't know when
> I will be having 5 times more users in the future. My concern right now is
> why all of my CPU cores are 100% busy and load is around 20.
> Thanks
> Paras.
>

you can use the FPM slowlog feature to trace all your long running requests.