Alright. I'm going to make a page, but for SVN/5.3+ I'm going to link
to that page, and put down something about the patch is not supported
by the PHP team, don't report a bug there. (A little bit of a triage)


On Sat, Apr 17, 2010 at 1:37 PM, Antony Dovgal <tony@daylessday.org> wrote:
> On 18.04.2010 00:30, Michael Shadle wrote:
>> I'd like to add a section on the site about how to report a bug.
>
> We already have that, no need for a duplicate:
> http://bugs.php.net/how-to-report.php
>
> A category for FPM bugs has been added recently, too.
>
> --
> Wbr,
> Antony Dovgal
> ---
> http://pinba.org - realtime statistics for PHP
>
>
> --
> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en
>

http://php-fpm.org/bugs/

Let me know if you think anything is insufficient.

On Sat, Apr 17, 2010 at 1:44 PM, Michael Shadle <mike503@gmail.com> wrote:
> Alright. I'm going to make a page, but for SVN/5.3+ I'm going to link
> to that page, and put down something about the patch is not supported
> by the PHP team, don't report a bug there. (A little bit of a triage)


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

> > Without execution time limit nothing is logged/output to stderr - just
> > the basic notices, child start, child termination on requests limit
> > etc.
>
> and did the problem appear ?

Yes, it did.



--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/17 Michael Shadle <mike503@gmail.com>:
> http://php-fpm.org/bugs/
>
> Let me know if you think anything is insufficient.
>

Only the SVN Version is fully supported !

> On Sat, Apr 17, 2010 at 1:44 PM, Michael Shadle <mike503@gmail.com> wrote:
>> Alright. I'm going to make a page, but for SVN/5.3+ I'm going to link
>> to that page, and put down something about the patch is not supported
>> by the PHP team, don't report a bug there. (A little bit of a triage)
>
>
> --
> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en
>

2010/4/17 Michael Shadle <mike503@gmail.com>:
> Unless Jerome disagrees since he put it in place I would still call the
> apache-like or dynamic process manager experimental. Use static and adjust
> children appropriately.

yes it is experimental ! which doesn't mean it doesn't work or is buggy :)


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/17 c2h5oh <maciej.lisiewski@gmail.com>:
>
>> > Without execution time limit nothing is logged/output to stderr - just
>> > the basic notices, child start, child termination on requests limit
>> > etc.
>>
>> and did the problem appear ?
>
> Yes, it did.

I'll say the same as tony:


If that doesn't work, provide a GDB backtrace at last!
If you don't know how to do it, read this:
http://bugs.php.net/bugs-generating-backtrace.php

Please, always start with the backtrace whenever you have any
segfaults, it saves a lot of time and traffic.


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

Yeah but it also means it's brand new and not necessarily 100% :)

I wish I had 5.3 code I supported so I could try it out myself!


On Apr 18, 2010, at 2:38 AM, Jérôme Loyet <ml@fatbsd.com> wrote:

> 2010/4/17 Michael Shadle <mike503@gmail.com>:
>> Unless Jerome disagrees since he put it in place I would still call
>> the
>> apache-like or dynamic process manager experimental. Use static and
>> adjust
>> children appropriately.
>
> yes it is experimental ! which doesn't mean it doesn't work or is
> buggy :)
>
>
> --
> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

we seem to do a "best we can do" help on the patch still typically.
And we should - 5.2 is very prevalent still

On Apr 18, 2010, at 2:34 AM, Jérôme Loyet <ml@fatbsd.com> wrote:

> 2010/4/17 Michael Shadle <mike503@gmail.com>:
>> http://php-fpm.org/bugs/
>>
>> Let me know if you think anything is insufficient.
>>
>
> Only the SVN Version is fully supported !
>
>> On Sat, Apr 17, 2010 at 1:44 PM, Michael Shadle <mike503@gmail.com>
>> wrote:
>>> Alright. I'm going to make a page, but for SVN/5.3+ I'm going to
>>> link
>>> to that page, and put down something about the patch is not
>>> supported
>>> by the PHP team, don't report a bug there. (A little bit of a
>>> triage)
>>
>>
>> --
>> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en
>>

> If that doesn't work, provide a GDB backtrace at last!
> If you don't know how to do it, read this:http://bugs.php.net/bugs-generating-backtrace.php
>
> Please, always start with the backtrace whenever you have any
> segfaults, it saves a lot of time and traffic.

As I wrote earlier I wasn't sure if segfaults were related to the
issue and as the time goes I am more and more sure they aren't - last
5 incidents happened without a segfault being logged. There is nothing
to backtrace.
So to follow the bug reporting guide:

1. What happens:
From time to time (2-4 times a day) PHP-FPM processes pool becomes
less and less responsive to finally grind to full halt with 502/504
errors thrown by nginx. This is not related to cpu/io/memory spike in
any way - each and every time I have seen it happen there was plenty
of free RAM, almost no IO load, and CPU load under 50% - and as the
responsiveness gets worse server load gets lower.
With PHP-FPM logging level set to debug NOTHING out of ordinary is
logged in PHP-FPM log. If PHP-FPM is run not daemonized I get
[WARNING] fpm_request_check_timed_out() if request_slowlog_timeout /
request_terminate_timeout is set or absolutely nothing out of ordinary
if left at 0.
Nginx logs connect() to unix:/var/run/php-fpm/php.sock failed (11:
Resource temporarily unavailable) while connecting to upstream
Sometimes, rarely, PHP-FPM pool regains responsiveness over time
(30-60min+), but mostly it has to be restarted to work again.

2. Latest version:
- I can't use 5.3.x, because some of the PHP code running on my server
is not 5.3.x ready yet - so I'm stuck with 5.2.x for now
- I have first noticed the problem with PHP 5.2.12 + PHP-FPM patch
- I have updated to PHP 5.2.13 + PHP-FPM patch ( php-5.2.13-
fpm-0.5.13 ) - made no difference.

3. System info:
- Debian 5.0 Lenny
- PHP 5.2.13, compiled with '--enable-bcmath' '--with-gd' '--with-
jpeg-dir=/usr/lib' '--with-zlib' '--with-mysql' '--enable-gd-native-
ttf' '--with-png-dir' '--with-ttf' '--with-pear' '--with-openssl' '--
with-curl' '--with-imap-ssl' '--enable-fastcgi' '--with-mcrypt' '--
with-mhash' '--enable-zip' '--enable-ftp' '--enable-magic-quotes' '--
enable-mbstring' '--enable-exif' '--with-bz2' '--enable-force-cgi-
redirect' '--with-config-file-path=/etc/php5' '--with-freetype-dir' '--
with-gettext' '--enable-pdo' '--with-pdo-mysql' '--with-pdo-pgsql' '--
with-pgsql' '--enable-discard-path' '--enable-fpm' '--with-imap' '--
with-kerberos' '--enable-calendar' '--enable-cli'


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/19 c2h5oh <maciej.lisiewski@gmail.com>:
>
>> If that doesn't work, provide a GDB backtrace at last!
>> If you don't know how to do it, read this:http://bugs.php.net/bugs-generating-backtrace.php
>>
>> Please, always start with the backtrace whenever you have any
>> segfaults, it saves a lot of time and traffic.
>
> As I wrote earlier I wasn't sure if segfaults were related to the
> issue and as the time goes I am more and more sure they aren't - last
> 5 incidents happened without a segfault being logged. There is nothing
> to backtrace.
> So to follow the bug reporting guide:
>
> 1. What happens:
> From time to time (2-4 times a day) PHP-FPM processes pool becomes
> less and less responsive to finally grind to full halt with 502/504
> errors thrown by nginx. This is not related to cpu/io/memory spike in
> any way - each and every time I have seen it happen there was plenty
> of free RAM, almost no IO load, and CPU load under 50% - and as the
> responsiveness gets worse server load gets lower.
> With PHP-FPM logging level set to debug NOTHING out of ordinary is
> logged in PHP-FPM log. If PHP-FPM is run not daemonized I get
> [WARNING] fpm_request_check_timed_out() if request_slowlog_timeout /
> request_terminate_timeout is set or absolutely nothing out of ordinary
> if left at 0.
> Nginx logs connect() to unix:/var/run/php-fpm/php.sock failed (11:
> Resource temporarily unavailable) while connecting to upstream
> Sometimes, rarely, PHP-FPM pool regains responsiveness over time
> (30-60min+), but mostly it has to be restarted to work again.
>
> 2. Latest version:
> - I can't use 5.3.x, because some of the PHP code running on my server
> is not 5.3.x ready yet - so I'm stuck with 5.2.x for now
> - I have first noticed the problem with PHP 5.2.12 + PHP-FPM patch
> - I have updated to PHP 5.2.13 + PHP-FPM patch ( php-5.2.13-
> fpm-0.5.13 ) - made no difference.
>
> 3. System info:
> - Debian 5.0 Lenny
> - PHP 5.2.13, compiled with  '--enable-bcmath' '--with-gd' '--with-
> jpeg-dir=/usr/lib' '--with-zlib' '--with-mysql' '--enable-gd-native-
> ttf' '--with-png-dir' '--with-ttf' '--with-pear' '--with-openssl' '--
> with-curl' '--with-imap-ssl' '--enable-fastcgi' '--with-mcrypt' '--
> with-mhash' '--enable-zip' '--enable-ftp' '--enable-magic-quotes' '--
> enable-mbstring' '--enable-exif' '--with-bz2' '--enable-force-cgi-
> redirect' '--with-config-file-path=/etc/php5' '--with-freetype-dir' '--
> with-gettext' '--enable-pdo' '--with-pdo-mysql' '--with-pdo-pgsql' '--
> with-pgsql' '--enable-discard-path' '--enable-fpm' '--with-imap' '--
> with-kerberos' '--enable-calendar' '--enable-cli'
>

it seems that your php scripts are running out of time and are
consuming the max_children processes. When you have bad response time,
you can count the number of fpm processes and see if it's the same as
max_children + 1 (1= the master process).

++ Jerome


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

Maybe start it with one child and start or attach strace to it. It may
give you some ideas as to slow downs when they occur. Bed return
values from system calls etc. This approach always yields some useful
information for me in these situations.

Dave.

On Apr 19, 9:42 am, Jérôme Loyet <m...@fatbsd.com> wrote:
> 2010/4/19 c2h5oh <maciej.lisiew...@gmail.com>:
>
>
>
>
>
>
>
> >> If that doesn't work, provide a GDB backtrace at last!
> >> If you don't know how to do it, read this:http://bugs.php.net/bugs-generating-backtrace.php
>
> >> Please, always start with the backtrace whenever you have any
> >> segfaults, it saves a lot of time and traffic.
>
> > As I wrote earlier I wasn't sure if segfaults were related to the
> > issue and as the time goes I am more and more sure they aren't - last
> > 5 incidents happened without a segfault being logged. There is nothing
> > to backtrace.
> > So to follow the bug reporting guide:
>
> > 1. What happens:
> > From time to time (2-4 times a day) PHP-FPM processes pool becomes
> > less and less responsive to finally grind to full halt with 502/504
> > errors thrown by nginx. This is not related to cpu/io/memory spike in
> > any way - each and every time I have seen it happen there was plenty
> > of free RAM, almost no IO load, and CPU load under 50% - and as the
> > responsiveness gets worse server load gets lower.
> > With PHP-FPM logging level set to debug NOTHING out of ordinary is
> > logged in PHP-FPM log. If PHP-FPM is run not daemonized I get
> > [WARNING] fpm_request_check_timed_out() if request_slowlog_timeout /
> > request_terminate_timeout is set or absolutely nothing out of ordinary
> > if left at 0.
> > Nginx logs connect() to unix:/var/run/php-fpm/php.sock failed (11:
> > Resource temporarily unavailable) while connecting to upstream
> > Sometimes, rarely, PHP-FPM pool regains responsiveness over time
> > (30-60min+), but mostly it has to be restarted to work again.
>
> > 2. Latest version:
> > - I can't use 5.3.x, because some of the PHP code running on my server
> > is not 5.3.x ready yet - so I'm stuck with 5.2.x for now
> > - I have first noticed the problem with PHP 5.2.12 + PHP-FPM patch
> > - I have updated to PHP 5.2.13 + PHP-FPM patch ( php-5.2.13-
> > fpm-0.5.13 ) - made no difference.
>
> > 3. System info:
> > - Debian 5.0 Lenny
> > - PHP 5.2.13, compiled with  '--enable-bcmath' '--with-gd' '--with-
> > jpeg-dir=/usr/lib' '--with-zlib' '--with-mysql' '--enable-gd-native-
> > ttf' '--with-png-dir' '--with-ttf' '--with-pear' '--with-openssl' '--
> > with-curl' '--with-imap-ssl' '--enable-fastcgi' '--with-mcrypt' '--
> > with-mhash' '--enable-zip' '--enable-ftp' '--enable-magic-quotes' '--
> > enable-mbstring' '--enable-exif' '--with-bz2' '--enable-force-cgi-
> > redirect' '--with-config-file-path=/etc/php5' '--with-freetype-dir' '--
> > with-gettext' '--enable-pdo' '--with-pdo-mysql' '--with-pdo-pgsql' '--
> > with-pgsql' '--enable-discard-path' '--enable-fpm' '--with-imap' '--
> > with-kerberos' '--enable-calendar' '--enable-cli'
>
> it seems that your php scripts are running out of time and are
> consuming the max_children processes. When you have bad response time,
> you can count the number of fpm processes and see if it's the same as
> max_children + 1 (1= the master process).
>
> ++ Jerome
>
> --
> Subscription settings:http://groups.google.com/group/highload-php-en/subscribe?hl=en

> it seems that your php scripts are running out of time and are
> consuming the max_children processes. When you have bad response time,
> you can count the number of fpm processes and see if it's the same as
> max_children + 1 (1= the master process).

yes, it is.


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
> Maybe start it with one child and start or attach strace to it. It may
> give you some ideas as to slow downs when they occur. Bed return
> values from system calls etc.  This approach always yields some useful
> information for me in these situations.

Waiting for next incident to do just that - will get back to you.


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/19 c2h5oh <maciej.lisiewski@gmail.com>:
>
>
> On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
>> Maybe start it with one child and start or attach strace to it. It may
>> give you some ideas as to slow downs when they occur. Bed return
>> values from system calls etc.  This approach always yields some useful
>> information for me in these situations.
>
> Waiting for next incident to do just that - will get back to you.

You can set
- request_slowlog_timeout to 5s
- slowlog to /tmp/slowlog.log

and if a request takes more than 5s, a ptrace of the process will be
written into /tmp/slowlog:

Apr 17 18:20:31.713958 [pool test] pid 25163
script_filename = /home/www/test/docs/wait10.php
[0x8a6c4020] sleep() /home/www/test/docs/wait10.php:4

it could help you debugging.

btw, for me it's not a FPM problem :)


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
> Maybe start it with one child and start or attach strace to it. It may
> give you some ideas as to slow downs when they occur. Bed return
> values from system calls etc.  This approach always yields some useful
> information for me in these situations.

I've got

restart_syscall(<... resuming interrupted call ...>

hanging for each and every child process


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/19 c2h5oh <maciej.lisiewski@gmail.com>:
>
>
> On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
>> Maybe start it with one child and start or attach strace to it. It may
>> give you some ideas as to slow downs when they occur. Bed return
>> values from system calls etc.  This approach always yields some useful
>> information for me in these situations.
>
> I've got
>
> restart_syscall(<... resuming interrupted call ...>
>
> hanging for each and every child process

can you try to have a stack trace please ?

gdb /path/to/php-fpm_binary pid_of_child
bt

++ jerome


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

> can you try to have a stack trace please ?
>
> gdb /path/to/php-fpm_binary pid_of_child
> bt
>

Sorry about the delay, heres gdb output:

Reading symbols from /lib/i686/cmov/libcrypt.so.1...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libcrypt-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libcrypt.so.1
Reading symbols from /usr/lib/libc-client.so.2002edebian...done.
Loaded symbols for /usr/lib/libc-client.so.2002edebian
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libpq.so.5...done.
Loaded symbols for /usr/lib/libpq.so.5
Reading symbols from /lib/i686/cmov/librt.so.1...Reading symbols from /
usr/lib/debug/lib/i686/cmov/librt-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/librt.so.1
Reading symbols from /usr/lib/libmhash.so.2...done.
Loaded symbols for /usr/lib/libmhash.so.2
Reading symbols from /usr/lib/libmcrypt.so.4...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /lib/libbz2.so.1.0...done.
Loaded symbols for /lib/libbz2.so.1.0
Reading symbols from /lib/i686/cmov/libresolv.so.2...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libresolv-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libresolv.so.2
Reading symbols from /lib/i686/cmov/libdl.so.2...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libdl-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /lib/i686/cmov/libnsl.so.1...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libnsl-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnsl.so.1
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libcurl.so.4...Reading symbols from /usr/
lib/debug/usr/lib/libcurl.so.4.1.0...done.
done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /usr/lib/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient.so.15
Reading symbols from /lib/i686/cmov/libpthread.so.0...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libpthread-2.7.so...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb735f6d0 (LWP 14326)]
done.
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libm.so.6...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libm-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libm.so.6
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/i686/cmov/libc.so.6...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libc-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/ld-linux.so.2...Reading symbols from /usr/
lib/debug/lib/ld-2.7.so...done.
done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/libssh2.so.1...done.
Loaded symbols for /usr/lib/libssh2.so.1
Reading symbols from /usr/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/libtasn1.so.3...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/xcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/xcache.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcache.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcached.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcached.so
Reading symbols from /usr/local/lib/libmemcached.so.2...done.
Loaded symbols for /usr/local/lib/libmemcached.so.2
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/imagick.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/imagick.so
Reading symbols from /usr/lib/libWand.so.10...done.
Loaded symbols for /usr/lib/libWand.so.10
Reading symbols from /usr/lib/libMagick.so.10...done.
Loaded symbols for /usr/lib/libMagick.so.10
Reading symbols from /usr/lib/liblcms.so.1...done.
Loaded symbols for /usr/lib/liblcms.so.1
Reading symbols from /usr/lib/libtiff.so.4...done.
Loaded symbols for /usr/lib/libtiff.so.4
Reading symbols from /usr/lib/libfontconfig.so.1...done.
Loaded symbols for /usr/lib/libfontconfig.so.1
Reading symbols from /usr/lib/libXext.so.6...done.
Loaded symbols for /usr/lib/libXext.so.6
Reading symbols from /usr/lib/libXt.so.6...done.
Loaded symbols for /usr/lib/libXt.so.6
Reading symbols from /usr/lib/libSM.so.6...done.
Loaded symbols for /usr/lib/libSM.so.6
Reading symbols from /usr/lib/libICE.so.6...done.
Loaded symbols for /usr/lib/libICE.so.6
Reading symbols from /usr/lib/libX11.so.6...done.
Loaded symbols for /usr/lib/libX11.so.6
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libXau.so.6...done.
Loaded symbols for /usr/lib/libXau.so.6
Reading symbols from /usr/lib/libxcb-xlib.so.0...done.
Loaded symbols for /usr/lib/libxcb-xlib.so.0
Reading symbols from /usr/lib/libxcb.so.1...done.
Loaded symbols for /usr/lib/libxcb.so.1
Reading symbols from /usr/lib/libXdmcp.so.6...done.
Loaded symbols for /usr/lib/libXdmcp.so.6
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/mailparse.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/mailparse.so
Reading symbols from /lib/i686/cmov/libnss_files.so.2...Reading
symbols from /usr/lib/debug/lib/i686/cmov/libnss_files-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_files.so.2
Reading symbols from /lib/i686/cmov/libnss_compat.so.2...Reading
symbols from /usr/lib/debug/lib/i686/cmov/libnss_compat-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_compat.so.2
Reading symbols from /lib/i686/cmov/libnss_nis.so.2...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libnss_nis-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_nis.so.2
0xb7f7f424 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f7f424 in __kernel_vsyscall ()
#1 0xb7809083 in __read_nocancel () from /lib/i686/cmov/libpthread.so.
0
#2 0xb78899d8 in vio_read () from /usr/lib/libmysqlclient.so.15
#3 0xb7889a2b in vio_read_buff () from /usr/lib/libmysqlclient.so.15
#4 0xb788ad19 in ?? () from /usr/lib/libmysqlclient.so.15
#5 0x08831288 in ?? ()
#6 0x08835a20 in ?? ()
#7 0x00000004 in ?? ()
#8 0x0884a918 in ?? ()
#9 0x00000000 in ?? ()


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/26 c2h5oh <maciej.lisiewski@gmail.com>:
>> can you try to have a stack trace please ?
>>
>> gdb /path/to/php-fpm_binary pid_of_child
>> bt
>>
>
> Sorry about the delay, heres gdb output:

can you remove all extensions and try again please ?

>
> Reading symbols from /lib/i686/cmov/libcrypt.so.1...Reading symbols
> from /usr/lib/debug/lib/i686/cmov/libcrypt-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libcrypt.so.1
> Reading symbols from /usr/lib/libc-client.so.2002edebian...done.
> Loaded symbols for /usr/lib/libc-client.so.2002edebian
> Reading symbols from /usr/lib/libz.so.1...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /usr/lib/libpq.so.5...done.
> Loaded symbols for /usr/lib/libpq.so.5
> Reading symbols from /lib/i686/cmov/librt.so.1...Reading symbols from /
> usr/lib/debug/lib/i686/cmov/librt-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/librt.so.1
> Reading symbols from /usr/lib/libmhash.so.2...done.
> Loaded symbols for /usr/lib/libmhash.so.2
> Reading symbols from /usr/lib/libmcrypt.so.4...done.
> Loaded symbols for /usr/lib/libmcrypt.so.4
> Reading symbols from /usr/lib/libltdl.so.3...done.
> Loaded symbols for /usr/lib/libltdl.so.3
> Reading symbols from /lib/libpam.so.0...done.
> Loaded symbols for /lib/libpam.so.0
> Reading symbols from /usr/lib/libfreetype.so.6...done.
> Loaded symbols for /usr/lib/libfreetype.so.6
> Reading symbols from /usr/lib/libpng12.so.0...done.
> Loaded symbols for /usr/lib/libpng12.so.0
> Reading symbols from /usr/lib/libjpeg.so.62...done.
> Loaded symbols for /usr/lib/libjpeg.so.62
> Reading symbols from /lib/libbz2.so.1.0...done.
> Loaded symbols for /lib/libbz2.so.1.0
> Reading symbols from /lib/i686/cmov/libresolv.so.2...Reading symbols
> from /usr/lib/debug/lib/i686/cmov/libresolv-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libresolv.so.2
> Reading symbols from /lib/i686/cmov/libdl.so.2...Reading symbols from /
> usr/lib/debug/lib/i686/cmov/libdl-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libdl.so.2
> Reading symbols from /lib/i686/cmov/libnsl.so.1...Reading symbols
> from /usr/lib/debug/lib/i686/cmov/libnsl-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libnsl.so.1
> Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /usr/lib/libgssapi_krb5.so.2
> Reading symbols from /usr/lib/libkrb5.so.3...done.
> Loaded symbols for /usr/lib/libkrb5.so.3
> Reading symbols from /usr/lib/libk5crypto.so.3...done.
> Loaded symbols for /usr/lib/libk5crypto.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
> Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
> Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
> Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
> Reading symbols from /usr/lib/libcurl.so.4...Reading symbols from /usr/
> lib/debug/usr/lib/libcurl.so.4.1.0...done.
> done.
> Loaded symbols for /usr/lib/libcurl.so.4
> Reading symbols from /usr/lib/libmysqlclient.so.15...done.
> Loaded symbols for /usr/lib/libmysqlclient.so.15
> Reading symbols from /lib/i686/cmov/libpthread.so.0...Reading symbols
> from /usr/lib/debug/lib/i686/cmov/libpthread-2.7.so...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0xb735f6d0 (LWP 14326)]
> done.
> Loaded symbols for /lib/i686/cmov/libpthread.so.0
> Reading symbols from /lib/i686/cmov/libm.so.6...Reading symbols from /
> usr/lib/debug/lib/i686/cmov/libm-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libm.so.6
> Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2
> Reading symbols from /lib/i686/cmov/libc.so.6...Reading symbols from /
> usr/lib/debug/lib/i686/cmov/libc-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libc.so.6
> Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
> Loaded symbols for /usr/lib/libldap_r-2.4.so.2
> Reading symbols from /lib/ld-linux.so.2...Reading symbols from /usr/
> lib/debug/lib/ld-2.7.so...done.
> done.
> Loaded symbols for /lib/ld-linux.so.2
> Reading symbols from /usr/lib/libkrb5support.so.0...done.
> Loaded symbols for /usr/lib/libkrb5support.so.0
> Reading symbols from /lib/libkeyutils.so.1...done.
> Loaded symbols for /lib/libkeyutils.so.1
> Reading symbols from /usr/lib/libidn.so.11...done.
> Loaded symbols for /usr/lib/libidn.so.11
> Reading symbols from /usr/lib/libssh2.so.1...done.
> Loaded symbols for /usr/lib/libssh2.so.1
> Reading symbols from /usr/lib/liblber-2.4.so.2...done.
> Loaded symbols for /usr/lib/liblber-2.4.so.2
> Reading symbols from /usr/lib/libsasl2.so.2...done.
> Loaded symbols for /usr/lib/libsasl2.so.2
> Reading symbols from /usr/lib/libgnutls.so.26...done.
> Loaded symbols for /usr/lib/libgnutls.so.26
> Reading symbols from /usr/lib/libgcrypt.so.11...done.
> Loaded symbols for /usr/lib/libgcrypt.so.11
> Reading symbols from /usr/lib/libgpg-error.so.0...done.
> Loaded symbols for /usr/lib/libgpg-error.so.0
> Reading symbols from /usr/lib/libtasn1.so.3...done.
> Loaded symbols for /usr/lib/libtasn1.so.3
> Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/xcache.so...done.
> Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/xcache.so
> Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/memcache.so...done.
> Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/memcache.so
> Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/memcached.so...done.
> Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/memcached.so
> Reading symbols from /usr/local/lib/libmemcached.so.2...done.
> Loaded symbols for /usr/local/lib/libmemcached.so.2
> Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/imagick.so...done.
> Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/imagick.so
> Reading symbols from /usr/lib/libWand.so.10...done.
> Loaded symbols for /usr/lib/libWand.so.10
> Reading symbols from /usr/lib/libMagick.so.10...done.
> Loaded symbols for /usr/lib/libMagick.so.10
> Reading symbols from /usr/lib/liblcms.so.1...done.
> Loaded symbols for /usr/lib/liblcms.so.1
> Reading symbols from /usr/lib/libtiff.so.4...done.
> Loaded symbols for /usr/lib/libtiff.so.4
> Reading symbols from /usr/lib/libfontconfig.so.1...done.
> Loaded symbols for /usr/lib/libfontconfig.so.1
> Reading symbols from /usr/lib/libXext.so.6...done.
> Loaded symbols for /usr/lib/libXext.so.6
> Reading symbols from /usr/lib/libXt.so.6...done.
> Loaded symbols for /usr/lib/libXt.so.6
> Reading symbols from /usr/lib/libSM.so.6...done.
> Loaded symbols for /usr/lib/libSM.so.6
> Reading symbols from /usr/lib/libICE.so.6...done.
> Loaded symbols for /usr/lib/libICE.so.6
> Reading symbols from /usr/lib/libX11.so.6...done.
> Loaded symbols for /usr/lib/libX11.so.6
> Reading symbols from /usr/lib/libexpat.so.1...done.
> Loaded symbols for /usr/lib/libexpat.so.1
> Reading symbols from /usr/lib/libXau.so.6...done.
> Loaded symbols for /usr/lib/libXau.so.6
> Reading symbols from /usr/lib/libxcb-xlib.so.0...done.
> Loaded symbols for /usr/lib/libxcb-xlib.so.0
> Reading symbols from /usr/lib/libxcb.so.1...done.
> Loaded symbols for /usr/lib/libxcb.so.1
> Reading symbols from /usr/lib/libXdmcp.so.6...done.
> Loaded symbols for /usr/lib/libXdmcp.so.6
> Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/mailparse.so...done.
> Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
> zts-20060613/mailparse.so
> Reading symbols from /lib/i686/cmov/libnss_files.so.2...Reading
> symbols from /usr/lib/debug/lib/i686/cmov/libnss_files-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libnss_files.so.2
> Reading symbols from /lib/i686/cmov/libnss_compat.so.2...Reading
> symbols from /usr/lib/debug/lib/i686/cmov/libnss_compat-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libnss_compat.so.2
> Reading symbols from /lib/i686/cmov/libnss_nis.so.2...Reading symbols
> from /usr/lib/debug/lib/i686/cmov/libnss_nis-2.7.so...done.
> done.
> Loaded symbols for /lib/i686/cmov/libnss_nis.so.2
> 0xb7f7f424 in __kernel_vsyscall ()
> (gdb) bt
> #0  0xb7f7f424 in __kernel_vsyscall ()
> #1  0xb7809083 in __read_nocancel () from /lib/i686/cmov/libpthread.so.
> 0
> #2  0xb78899d8 in vio_read () from /usr/lib/libmysqlclient.so.15
> #3  0xb7889a2b in vio_read_buff () from /usr/lib/libmysqlclient.so.15
> #4  0xb788ad19 in ?? () from /usr/lib/libmysqlclient.so.15
> #5  0x08831288 in ?? ()
> #6  0x08835a20 in ?? ()
> #7  0x00000004 in ?? ()
> #8  0x0884a918 in ?? ()
> #9  0x00000000 in ?? ()
>
>
> --
> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en
>

> can you remove all extensions and try again please ?

Wish I could - I am unable to force this error to happen (as I wrote
before I have even tried hammering it with AB for hours), therefore I
am down to running this on my production servers with 15s interval
monitoring. Since it's production imagick and memcached have to stay.
I can replace xcache with APC.

Sorry for making it extra difficult to pinpoint the error - If you can
offer some suggestions how to trigger the error on purpose I'm willing
to setup another box for just testing purpose.


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

2010/4/26 c2h5oh <maciej.lisiewski@gmail.com>:
>
>> can you remove all extensions and try again please ?
>
> Wish I could - I am unable to force this error to happen (as I wrote
> before I have even tried hammering it with AB for hours), therefore I
> am down to running this on my production servers with 15s interval
> monitoring. Since it's production imagick and memcached have to stay.
> I can replace xcache with APC.
>
> Sorry for making it extra difficult to pinpoint the error - If you can
> offer some suggestions how to trigger the error on purpose I'm willing
> to setup another box for just testing purpose.

Your stack trace shows a problem in the mysql library. But there is
not more informations about the previous calls. Maybe you can try to
compile PHP with --enable-debug to have a better stack trace. Or try
to use another mysql library ... I don't really know how to help you


>
>
> --
> Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en
>