I'm having some issues with nginx randomly throwing out 502 gateway
timeout, the error log contains only this:

Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
(PEEKDATA) failed: Input/output error (5)

There is literally no information on google about that error message.

I run nginx 0.8.13 and PHP is 5.2.8 compiled with php-fpm

I use a named socket to pass to fastcgi: fastcgi_pass unix:/tmp/
php.socket;

my php-fpm configuration: http://pastie.org/606356

Additionally, nginx logs this error from php:

2009/09/04 22:11:23 [error] 16990#0: *1240096 connect() to unix:/tmp/
php.socket failed (11: Resource temporarily unavailable) while
connecting to upstream

So for some reason the socket is randomly unavilable.

On Fri, Sep 4, 2009 at 10:15 PM, Martin Fjordvald<nofearroxo@gmail.com> wrote:
>
> I'm having some issues with nginx randomly throwing out 502 gateway
> timeout, the error log contains only this:
>
> Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
> (PEEKDATA) failed: Input/output error (5)
>
> There is literally no information on google about that error message.

I personally don't know this, but have seen prior reports about 502's.
Might help to search again on keywords FPM + 502 error
+ "few minutes" / "occasionally" / "sometimes" / "80%-90%".

You should pick something up.


dreamcat4
dreamcat4@gmail.com

The search is horrible to be honest, searching for "socket" or
"gateway" gave no results, but browsing the topic list gave this
thread which discusses the exact same issue.

http://groups.google.com/group/highload-php-en/browse_thread/thread/72987134573199ec#

I guess I'll switch over to a TCP socket and deal with the issues
there instead.

On Sep 5, 12:31 am, dreamcat four <dreamc...@gmail.com> wrote:
> On Fri, Sep 4, 2009 at 10:15 PM, Martin Fjordvald<nofearr...@gmail.com> wrote:
>
> > I'm having some issues with nginx randomly throwing out 502 gateway
> > timeout, the error log contains only this:
>
> > Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
> > (PEEKDATA) failed: Input/output error (5)
>
> > There is literally no information on google about that error message.
>
> I personally don't know this, but have seen prior reports about 502's.
> Might help to search again on keywords FPM + 502 error
> + "few minutes" / "occasionally" / "sometimes" / "80%-90%".
>
> You should pick something up.
>
> dreamcat4
> dreamc...@gmail.com

changing to TCP would have been my suggestion, to see if it helps.

On Fri, Sep 4, 2009 at 4:07 PM, Martin Fjordvald<nofearroxo@gmail.com> wrote:
>
> The search is horrible to be honest, searching for "socket" or
> "gateway" gave no results, but browsing the topic list gave this
> thread which discusses the exact same issue.
>
> http://groups.google.com/group/highload-php-en/browse_thread/thread/72987134573199ec#
>
> I guess I'll switch over to a TCP socket and deal with the issues
> there instead.
>
> On Sep 5, 12:31 am, dreamcat four <dreamc...@gmail.com> wrote:
>> On Fri, Sep 4, 2009 at 10:15 PM, Martin Fjordvald<nofearr...@gmail.com> wrote:
>>
>> > I'm having some issues with nginx randomly throwing out 502 gateway
>> > timeout, the error log contains only this:
>>
>> > Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
>> > (PEEKDATA) failed: Input/output error (5)
>>
>> > There is literally no information on google about that error message.
>>
>> I personally don't know this, but have seen prior reports about 502's.
>> Might help to search again on keywords FPM + 502 error
>> + "few minutes" / "occasionally" / "sometimes" / "80%-90%".
>>
>> You should pick something up.
>>
>> dreamcat4
>> dreamc...@gmail.com

Problem is TCP gives the same problems with random 504 gateway
timeouts and then "upstream timed out (110: Connection timed out)
while connecting to upstream" in the error log.

I've been through everything trying to figure out why it would
timeout, can't be database as there has only been 7700 slow queries
taking more than 1 second in the last 443 million queries. And MySQL
administrator only reports a lot of sleeping connections. So something
there was making the processes hang.

Guess named socket wasn't the solution, I'll probably try to upgrade
to PHP 5.3.0 and install latest php-fpm to see if that fixes it, if
that doesn't then revert back to spawn-fcgi or default PHP.

Appreciate the help, though.

On Sep 5, 1:18 am, Michael Shadle <mike...@gmail.com> wrote:
> changing to TCP would have been my suggestion, to see if it helps.
>
> On Fri, Sep 4, 2009 at 4:07 PM, Martin Fjordvald<nofearr...@gmail.com> wrote:
>
> > The search is horrible to be honest, searching for "socket" or
> > "gateway" gave no results, but browsing the topic list gave this
> > thread which discusses the exact same issue.
>
> >http://groups.google.com/group/highload-php-en/browse_thread/thread/7...
>
> > I guess I'll switch over to a TCP socket and deal with the issues
> > there instead.
>
> > On Sep 5, 12:31 am, dreamcat four <dreamc...@gmail.com> wrote:
> >> On Fri, Sep 4, 2009 at 10:15 PM, Martin Fjordvald<nofearr...@gmail.com> wrote:
>
> >> > I'm having some issues with nginx randomly throwing out 502 gateway
> >> > timeout, the error log contains only this:
>
> >> > Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
> >> > (PEEKDATA) failed: Input/output error (5)
>
> >> > There is literally no information on google about that error message.
>
> >> I personally don't know this, but have seen prior reports about 502's.
> >> Might help to search again on keywords FPM + 502 error
> >> + "few minutes" / "occasionally" / "sometimes" / "80%-90%".
>
> >> You should pick something up.
>
> >> dreamcat4
> >> dreamc...@gmail.com
>
>

На Saturday 05 September 2009 00:15:47 Martin Fjordvald написа:
> I'm having some issues with nginx randomly throwing out 502 gateway
> timeout, the error log contains only this:
>
> Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
> (PEEKDATA) failed: Input/output error (5)
>
> There is literally no information on google about that error message.
>
> I run nginx 0.8.13 and PHP is 5.2.8 compiled with php-fpm
>
> I use a named socket to pass to fastcgi: fastcgi_pass unix:/tmp/
> php.socket;
>
> my php-fpm configuration: http://pastie.org/606356

Hi,

I have solved my 502 issues by using more pools per server. I use 3 instead of
just one, it seems that there are some problems at high loads when using only
one socket. Moreover I use the feature "request_terminate_timeout" of fpm, I
set it to 30 seconds in my setup.

Momchil

--
PGP KeyID: 0x3118168B
Keyserver: pgp.mit.edu
Key fingerprint BB50 2983 0714 36DC D02E  158A E03D 56DA 3118 168B

I had that idea as well actually, could you show me your nginx
configuration for doing this? I was messing around a bit with it but
it seemed I had to switch to proxy_pass instead of fastcgi_pass and I
didn't really have the time to research proxy_pass properly.

On Sep 5, 12:48 pm, Momchil Ivanov <slogs...@gmail.com> wrote:
> На Saturday 05 September 2009 00:15:47 Martin Fjordvald написа:
>
> > I'm having some issues with nginx randomly throwing out 502 gateway
> > timeout, the error log contains only this:
>
> > Sep 04 19:29:26.777150 [ERROR] fpm_trace_get_long(), line 78: ptrace
> > (PEEKDATA) failed: Input/output error (5)
>
> > There is literally no information on google about that error message.
>
> > I run nginx 0.8.13 and PHP is 5.2.8 compiled with php-fpm
>
> > I use a named socket to pass to fastcgi: fastcgi_pass unix:/tmp/
> > php.socket;
>
> > my php-fpm configuration:http://pastie.org/606356
>
> Hi,
>
> I have solved my 502 issues by using more pools per server. I use 3 instead of
> just one, it seems that there are some problems at high loads when using only
> one socket. Moreover I use the feature "request_terminate_timeout" of fpm, I
> set it to 30 seconds in my setup.
>
> Momchil
>
> --
> PGP KeyID: 0x3118168B
> Keyserver: pgp.mit.edu
> Key fingerprint BB50 2983 0714 36DC D02E 158A E03D 56DA 3118 168B
>
> signature.asc
> < 1KViewDownload

На Sunday 06 September 2009 00:00:47 Martin Fjordvald написа:
> I had that idea as well actually, could you show me your nginx
> configuration for doing this? I was messing around a bit with it but
> it seemed I had to switch to proxy_pass instead of fastcgi_pass and I
> didn't really have the time to research proxy_pass properly.

It looks like this:

user nginx nginx;
worker_processes 8;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 10240;
}

http {
include mime.types;
default_type application/octet-stream;

access_log off;

## Timeouts
client_body_timeout 5;
client_header_timeout 5;
keepalive_timeout 0;
send_timeout 5;

## Server
server_name_in_redirect off;
server_tokens off;
ignore_invalid_headers on;
sendfile on;
client_header_buffer_size 2k;
large_client_header_buffers 10 16k;
client_max_body_size 10m;

## FastCGI
fastcgi_buffers 32 4k;
fastcgi_index index.php;
fastcgi_connect_timeout 2;
fastcgi_send_timeout 1;

## TCP options
tcp_nodelay on;
tcp_nopush on;

gzip on;
gzip_comp_level 1;
gzip_proxied any;
gzip_types text/plain text/html application/xml;
gzip_disable "MSIE [1-6]\.";
gzip_vary on;

## PHP pools

upstream phpfarm {
# omega
server 192.168.184.18:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.18:9901 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.18:9902 weight=100 max_fails=10
fail_timeout=5;
# lexus
server 192.168.184.15:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.15:9901 weight=100 max_fails=10
fail_timeout=5;
# scout
server 192.168.184.16:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.16:9901 weight=100 max_fails=10
fail_timeout=5;
# terran
server 192.168.184.12:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.12:9901 weight=100 max_fails=10
fail_timeout=5;
# gama
server 192.168.184.27:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.27:9901 weight=100 max_fails=10
fail_timeout=5;
# bebop
server 192.168.184.29:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.29:9901 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.29:9902 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.29:9903 weight=100 max_fails=10
fail_timeout=5;
# killer
server 192.168.184.31:9900 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.31:9901 weight=100 max_fails=10
fail_timeout=5;
server 192.168.184.31:9902 weight=100 max_fails=10
fail_timeout=5;
}

## Servers

server {
listen 80;
server_name domain.com;
root /www;
index index.php;

error_page 500 http://domain.com/500_error.html;
error_page 502 http://domain.com/502_error.html;
error_page 503 http://domain.com/503_error.html;

location ~ \.php$ {
fastcgi_pass phpfarm;
fastcgi_param SCRIPT_FILENAME
$document_root$fastcgi_script_name;
fastcgi_intercept_errors on;
fastcgi_read_timeout 120;
fastcgi_next_upstream error timeout invalid_header;
include fastcgi_params;
}
}
}


--
PGP KeyID: 0x3118168B
Keyserver: pgp.mit.edu
Key fingerprint BB50 2983 0714 36DC D02E  158A E03D 56DA 3118 168B

I don't see the need for multiple pools on each machine.

Just make one larger pool, and fastcgi_pass to it.

Also confirm that fastcgi_pass can used named locations (it probably
can) and you may have to prefix it with @ to work nicely (I am not
sure if that's 100% a requirement)

On Sat, Sep 5, 2009 at 3:45 PM, Momchil Ivanov<slogster@gmail.com> wrote:
> На Sunday 06 September 2009 00:00:47 Martin Fjordvald написа:
>> I had that idea as well actually, could you show me your nginx
>> configuration for doing this? I was messing around a bit with it but
>> it seemed I had to switch to proxy_pass instead of fastcgi_pass and I
>> didn't really have the time to research proxy_pass properly.
>
> It looks like this:
>
> user  nginx nginx;
> worker_processes  8;
>
> error_log       /var/log/nginx/error.log;
> pid             /var/run/nginx.pid;
>
> events {
>        worker_connections  10240;
> }
>
> http {
>        include         mime.types;
>        default_type    application/octet-stream;
>
>        access_log  off;
>
>        ## Timeouts
>        client_body_timeout     5;
>        client_header_timeout   5;
>        keepalive_timeout       0;
>        send_timeout            5;
>
>        ## Server
>        server_name_in_redirect off;
>        server_tokens           off;
>        ignore_invalid_headers  on;
>        sendfile                on;
>        client_header_buffer_size       2k;
>        large_client_header_buffers     10 16k;
>        client_max_body_size    10m;
>
>        ## FastCGI
>        fastcgi_buffers         32 4k;
>        fastcgi_index           index.php;
>        fastcgi_connect_timeout 2;
>        fastcgi_send_timeout 1;
>
>        ## TCP options
>        tcp_nodelay     on;
>        tcp_nopush      on;
>
>        gzip            on;
>        gzip_comp_level 1;
>        gzip_proxied    any;
>        gzip_types      text/plain text/html application/xml;
>        gzip_disable    "MSIE [1-6]\.";
>        gzip_vary       on;
>
>        ## PHP pools
>
>        upstream phpfarm {
>                # omega
>                server 192.168.184.18:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.18:9901 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.18:9902 weight=100 max_fails=10
> fail_timeout=5;
>                # lexus
>                server 192.168.184.15:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.15:9901 weight=100 max_fails=10
> fail_timeout=5;
>                # scout
>                server 192.168.184.16:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.16:9901 weight=100 max_fails=10
> fail_timeout=5;
>                # terran
>                server 192.168.184.12:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.12:9901 weight=100 max_fails=10
> fail_timeout=5;
>                # gama
>                server 192.168.184.27:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.27:9901 weight=100 max_fails=10
> fail_timeout=5;
>                # bebop
>                server 192.168.184.29:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.29:9901 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.29:9902 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.29:9903 weight=100 max_fails=10
> fail_timeout=5;
>                # killer
>                server 192.168.184.31:9900 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.31:9901 weight=100 max_fails=10
> fail_timeout=5;
>                server 192.168.184.31:9902 weight=100 max_fails=10
> fail_timeout=5;
>        }
>
>        ## Servers
>
>        server {
>                listen     80;
>                server_name domain.com;
>                root /www;
>                index index.php;
>
>                error_page      500  http://domain.com/500_error.html;
>                error_page      502  http://domain.com/502_error.html;
>                error_page      503  http://domain.com/503_error.html;
>
>                location ~ \.php$ {
>                        fastcgi_pass  phpfarm;
>                        fastcgi_param  SCRIPT_FILENAME
> $document_root$fastcgi_script_name;
>                        fastcgi_intercept_errors on;
>                        fastcgi_read_timeout 120;
>                        fastcgi_next_upstream error timeout invalid_header;
>                        include         fastcgi_params;
>                }
>        }
> }
>
>
> --
> PGP KeyID: 0x3118168B
> Keyserver: pgp.mit.edu
> Key fingerprint BB50 2983 0714 36DC D02E  158A E03D 56DA 3118 168B
>
>

На Sunday 06 September 2009 02:07:13 Michael Shadle написа:
> I don't see the need for multiple pools on each machine.
>
> Just make one larger pool, and fastcgi_pass to it.
>
> Also confirm that fastcgi_pass can used named locations (it probably
> can) and you may have to prefix it with @ to work nicely (I am not
> sure if that's 100% a requirement)

I've been using this config for almost a year now and I restart the fpm pools
every 2 hours. It works like a charm.

I cannot give you certain numbers as of why it is better to use more than one
socket per server. I can only speculate that in my setup with php 5.2.6 +
fpm, restarting the pools every 2 hours was a must. I haven't checked that
with 5.2.10 but I restart those too. And restarting pools automatically leads
to the increase of the number of pools in order to achieve smooth operation
of the whole setup. Otherwise you are left with say 1/5 less php processes
for about 5 secs when you restart a pool, so better have more pools -> loose
less processes per restart.

I can speculate that php 5.2.8 and 5.2.6 with fpm being used with nginx goes
to it's knees with only one socket per fpm server. You can try it yourself:
start one php pool of say 100-400 processes and try serving phpinfo(). I was
getting 50x errors at say 80-200 concurrent requests I think (I can only
speculate, I did test this more than a year ago and don't have any numbers
now, moreover I am too lazy to test it right now, but if you need any numbers
I might do it).

Momchil

--
PGP KeyID: 0x3118168B
Keyserver: pgp.mit.edu
Key fingerprint BB50 2983 0714 36DC D02E  158A E03D 56DA 3118 168B

Hi Momchil,

What's the method you use to restart the pools every 2 hours?


Best regards,

dreamcat4
dreamcat4@gmail.com