Nginx high load - php stops at some point

katmai

Nginx high load - php stops at some point
September 28, 2010 01:16PM

Hi guys,

I am running a very busy website. It's running a photo gallery ->
Gallery2.

The system is a Quad core Xeon, with 16 gb RAM, and separate drives
for OS/SQL/site, running FreeBSD 7.2 - latest patch level.

At some point during the day, pretty much random, the load on the
server spikes, and then php processing stops. I run nginx 0.8.52, and
php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
cache.

I have been debugging and i noticed that when the server reaches over
6000 active connections, then everything gets clogged and eventually i
had to build a script that would restart php/nginx when the load
starts spiking, which is not quite a nice solution. I don't like doing
patchwork like this, but i had to.

The behavior i noticed was when the php started dieing, then all php
processes would be in "pfault" or "sbwait" state.

If you need any more infos i will be happy to provide. i just want to
know what may be the cause of this.

Information:
1 - This is what nginx_status page shows when php is close to dieing.

Active connections: 5018
server accepts handled requests
11836 11836 28147
Reading: 49 Writing: 2103 Waiting: 2866

2 - my php-fpm.conf

<?xml version="1.0" ?>
<configuration>

All relative paths in this config are relative to php's
install prefix

<section name="global_options">

Pid file
<value name="pid_file">/usr/local/kphp/logs/php-
fpm.pid</value>

Error log file
<value name="error_log">/usr/local/kphp/logs/php-
fpm.log</value>

Log level
<value name="log_level">notice</value>

When this amount of php processes exited with SIGSEGV
or SIGBUS ...
<value name="emergency_restart_threshold">10</value>

... in a less than this interval of time, a graceful
restart will be initiated.
Useful to work around accidental curruptions in
accelerator's shared memory.
<value name="emergency_restart_interval">1m</value>

Time limit on waiting child's reaction on signals from
master
<value name="process_control_timeout">5s</value>

Set to 'no' to debug fpm
<value name="daemonize">yes</value>

</section>

<workers>

<section name="pool">

Name of pool. Used in logs and stats.
<value name="name">nginx</value>

Address to accept fastcgi requests on.
Valid syntax is 'ip.ad.re.ss:port' or just
'port' or '/path/to/unix/socket'
<value name="listen_address">127.0.0.1:8000</
value>

<value name="listen_options">

Set listen(2) backlog
<value name="backlog">-1</value>

Set permissions for unix socket, if
one used.
In Linux read/write permissions must
be set in order to allow connections from web server.
Many BSD-derrived systems allow
connections regardless of permissions.
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>

Additional php.ini defines, specific to this
pool of workers.
<value name="php_defines">


</value>

Unix user of processes
<value name="user">nobody</value>

Unix group of processes
<value name="group">nobody</value>

Process manager settings
<value name="pm">

Sets style of controling worker
process count.
Valid values are 'static' and 'apache-
like'
<value name="style">static</value>

Sets the limit on the number of
simultaneous requests that will be served.
Equivalent to Apache MaxClients
directive.
Equivalent to PHP_FCGI_CHILDREN
environment in original php.fcgi
Used with any pm_style.
<value name="max_children">256</value>

Settings group for 'apache-like' pm
style
<value name="apache_like">

Sets the number of server
processes created on startup.
Used only when 'apache-like'
pm_style is selected
<value name="StartServers">32</
value>

Sets the desired minimum
number of idle server processes.
Used only when 'apache-like'
pm_style is selected
<value
name="MinSpareServers">64</value>

Sets the desired maximum
number of idle server processes.
Used only when 'apache-like'
pm_style is selected
<value
name="MaxSpareServers">128</value>

</value>

</value>

The timeout (in seconds) for serving a single
request after which the worker process will be terminated
Should be used when 'max_execution_time' ini
option does not stop script execution for some reason
'0s' means 'off'
<value name="request_terminate_timeout">0s</
value>

The timeout (in seconds) for serving of single
request after which a php backtrace will be dumped to slow.log file
'0s' means 'off'
<value name="request_slowlog_timeout">10s</
value>

The log file for slow requests
<value name="slowlog">logs/slow.log</value>

Set open file desc rlimit
<value name="rlimit_files">8092</value>

Set max core size rlimit
<value name="rlimit_core">0</value>

Chroot to this directory at the start,
absolute path
<value name="chroot"></value>

Chdir to this directory at the start, absolute
path
<value name="chdir"></value>

Redirect workers' stdout and stderr into main
error log.
If not set, they will be redirected to /dev/
null, according to FastCGI specs
<value name="catch_workers_output">yes</value>

How much requests each process should execute
before respawn.
Useful to work around memory leaks in 3rd
party libraries.
For endless request processing please specify
0
Equivalent to PHP_FCGI_MAX_REQUESTS
<value name="max_requests">500</value>

Comma separated list of ipv4 addresses of
FastCGI clients that allowed to connect.
Equivalent to FCGI_WEB_SERVER_ADDRS
environment in original php.fcgi (5.2.2+)
Makes sense only with AF_INET listening
socket.
<value name="allowed_clients">127.0.0.1</
value>

Pass environment variables like
LD_LIBRARY_PATH
All $VARIABLEs are taken from current
environment
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</
value>
<value name="PATH">/usr/local/bin:/usr/
bin:/bin</value>
<value name="TMP">/tmp</value>
<value name="TMPDIR">/tmp</value>
<value name="TEMP">/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</
value>
<value name="MALLOC_CHECK_">2</value>
</value>

</section>

</workers>

</configuration>

3 - the relevant portion of nginx config:

user nobody;
worker_processes 64;
worker_rlimit_nofile 10240;

error_log logs/error.log;
pid logs/nginx.pid;

events {
worker_connections 8192;
use kqueue;
}

[...............................]

location ~* \.php$ {
fastcgi_pass 127.0.0.1:8000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
public_html$fastcgi_script_name;
fastcgi_param SCRIPT_NAME
$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PROTOCOL
$server_protocol;

fastcgi_buffers 64 64k;
fastcgi_buffer_size 64k;

Reply Quote

Jonathan Langevin

Re: Nginx high load - php stops at some point
September 28, 2010 01:20PM

I was experiencing similar issues recently.
Changed my opcode cache to APC, haven't seen the same problem since.

Try APC and see if you still have the same issues.

-Jon

On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:

> Hi guys,
>
> I am running a very busy website. It's running a photo gallery ->
> Gallery2.
>
> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>
> At some point during the day, pretty much random, the load on the
> server spikes, and then php processing stops. I run nginx 0.8.52, and
> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
> cache.
>
> I have been debugging and i noticed that when the server reaches over
> 6000 active connections, then everything gets clogged and eventually i
> had to build a script that would restart php/nginx when the load
> starts spiking, which is not quite a nice solution. I don't like doing
> patchwork like this, but i had to.
>
> The behavior i noticed was when the php started dieing, then all php
> processes would be in "pfault" or "sbwait" state.
>
> If you need any more infos i will be happy to provide. i just want to
> know what may be the cause of this.
>
> Information:
> 1 - This is what nginx_status page shows when php is close to dieing.
>
> Active connections: 5018
> server accepts handled requests
> 11836 11836 28147
> Reading: 49 Writing: 2103 Waiting: 2866
>
> 2 - my php-fpm.conf
>
>
> <?xml version="1.0" ?>
> <configuration>
>
> All relative paths in this config are relative to php's
> install prefix
>
> <section name="global_options">
>
> Pid file
> <value name="pid_file">/usr/local/kphp/logs/php-
> fpm.pid</value>
>
> Error log file
> <value name="error_log">/usr/local/kphp/logs/php-
> fpm.log</value>
>
> Log level
> <value name="log_level">notice</value>
>
> When this amount of php processes exited with SIGSEGV
> or SIGBUS ...
> <value name="emergency_restart_threshold">10</value>
>
> ... in a less than this interval of time, a graceful
> restart will be initiated.
> Useful to work around accidental curruptions in
> accelerator's shared memory.
> <value name="emergency_restart_interval">1m</value>
>
> Time limit on waiting child's reaction on signals from
> master
> <value name="process_control_timeout">5s</value>
>
> Set to 'no' to debug fpm
> <value name="daemonize">yes</value>
>
> </section>
>
> <workers>
>
> <section name="pool">
>
> Name of pool. Used in logs and stats.
> <value name="name">nginx</value>
>
> Address to accept fastcgi requests on.
> Valid syntax is 'ip.ad.re.ss:port' or just
> 'port' or '/path/to/unix/socket'
> <value name="listen_address">127.0.0.1:8000</
> value>
>
> <value name="listen_options">
>
> Set listen(2) backlog
> <value name="backlog">-1</value>
>
> Set permissions for unix socket, if
> one used.
> In Linux read/write permissions must
> be set in order to allow connections from web server.
> Many BSD-derrived systems allow
> connections regardless of permissions.
> <value name="owner"></value>
> <value name="group"></value>
> <value name="mode">0666</value>
> </value>
>
> Additional php.ini defines, specific to this
> pool of workers.
> <value name="php_defines">
> 
> 
> </value>
>
> Unix user of processes
> <value name="user">nobody</value>
>
> Unix group of processes
> <value name="group">nobody</value>
>
> Process manager settings
> <value name="pm">
>
> Sets style of controling worker
> process count.
> Valid values are 'static' and 'apache-
> like'
> <value name="style">static</value>
>
> Sets the limit on the number of
> simultaneous requests that will be served.
> Equivalent to Apache MaxClients
> directive.
> Equivalent to PHP_FCGI_CHILDREN
> environment in original php.fcgi
> Used with any pm_style.
> <value name="max_children">256</value>
>
> Settings group for 'apache-like' pm
> style
> <value name="apache_like">
>
> Sets the number of server
> processes created on startup.
> Used only when 'apache-like'
> pm_style is selected
> <value name="StartServers">32</
> value>
>
> Sets the desired minimum
> number of idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MinSpareServers">64</value>
>
> Sets the desired maximum
> number of idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MaxSpareServers">128</value>
>
> </value>
>
> </value>
>
> The timeout (in seconds) for serving a single
> request after which the worker process will be terminated
> Should be used when 'max_execution_time' ini
> option does not stop script execution for some reason
> '0s' means 'off'
> <value name="request_terminate_timeout">0s</
> value>
>
> The timeout (in seconds) for serving of single
> request after which a php backtrace will be dumped to slow.log file
> '0s' means 'off'
> <value name="request_slowlog_timeout">10s</
> value>
>
> The log file for slow requests
> <value name="slowlog">logs/slow.log</value>
>
> Set open file desc rlimit
> <value name="rlimit_files">8092</value>
>
> Set max core size rlimit
> <value name="rlimit_core">0</value>
>
> Chroot to this directory at the start,
> absolute path
> <value name="chroot"></value>
>
> Chdir to this directory at the start, absolute
> path
> <value name="chdir"></value>
>
> Redirect workers' stdout and stderr into main
> error log.
> If not set, they will be redirected to /dev/
> null, according to FastCGI specs
> <value name="catch_workers_output">yes</value>
>
> How much requests each process should execute
> before respawn.
> Useful to work around memory leaks in 3rd
> party libraries.
> For endless request processing please specify
> 0
> Equivalent to PHP_FCGI_MAX_REQUESTS
> <value name="max_requests">500</value>
>
> Comma separated list of ipv4 addresses of
> FastCGI clients that allowed to connect.
> Equivalent to FCGI_WEB_SERVER_ADDRS
> environment in original php.fcgi (5.2.2+)
> Makes sense only with AF_INET listening
> socket.
> <value name="allowed_clients">127.0.0.1</
> value>
>
> Pass environment variables like
> LD_LIBRARY_PATH
> All $VARIABLEs are taken from current
> environment
> <value name="environment">
> <value name="HOSTNAME">$HOSTNAME</
> value>
> <value name="PATH">/usr/local/bin:/usr/
> bin:/bin</value>
> <value name="TMP">/tmp</value>
> <value name="TMPDIR">/tmp</value>
> <value name="TEMP">/tmp</value>
> <value name="OSTYPE">$OSTYPE</value>
> <value name="MACHTYPE">$MACHTYPE</
> value>
> <value name="MALLOC_CHECK_">2</value>
> </value>
>
> </section>
>
> </workers>
>
> </configuration>
>
> 3 - the relevant portion of nginx config:
>
> user nobody;
> worker_processes 64;
> worker_rlimit_nofile 10240;
>
> error_log logs/error.log;
> pid logs/nginx.pid;
>
> events {
> worker_connections 8192;
> use kqueue;
> }
>
> [...............................]
>
> location ~* \.php$ {
> fastcgi_pass 127.0.0.1:8000;
> fastcgi_index index.php;
> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
> public_html$fastcgi_script_name;
> fastcgi_param SCRIPT_NAME
> $fastcgi_script_name;
> fastcgi_param REQUEST_URI $request_uri;
> fastcgi_param DOCUMENT_URI $document_uri;
> fastcgi_param DOCUMENT_ROOT $document_root;
> fastcgi_param QUERY_STRING $query_string;
> fastcgi_param REQUEST_METHOD $request_method;
> fastcgi_param CONTENT_TYPE $content_type;
> fastcgi_param CONTENT_LENGTH $content_length;
> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
> fastcgi_param SERVER_SOFTWARE nginx;
> fastcgi_param REMOTE_ADDR $remote_addr;
> fastcgi_param REMOTE_PORT $remote_port;
> fastcgi_param SERVER_ADDR $server_addr;
> fastcgi_param SERVER_PORT $server_port;
> fastcgi_param SERVER_NAME $server_name;
> fastcgi_param SERVER_PROTOCOL
> $server_protocol;
>
> fastcgi_buffers 64 64k;
> fastcgi_buffer_size 64k;
>

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 28, 2010 01:22PM

I tried APC prior to eAccelerator, and i had the same issues. It
actually happened more often for php to hang than now with eaccelerator.

On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
> I was experiencing similar issues recently.
> Changed my opcode cache to APC, haven't seen the same problem since.
>
> Try APC and see if you still have the same issues.
>
> -Jon
>
> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com
> <mailto:themediaplanet90210@gmail.com>> wrote:
>
> Hi guys,
>
> I am running a very busy website. It's running a photo gallery ->
> Gallery2.
>
> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>
> At some point during the day, pretty much random, the load on the
> server spikes, and then php processing stops. I run nginx 0.8.52, and
> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
> cache.
>
> I have been debugging and i noticed that when the server reaches over
> 6000 active connections, then everything gets clogged and eventually i
> had to build a script that would restart php/nginx when the load
> starts spiking, which is not quite a nice solution. I don't like doing
> patchwork like this, but i had to.
>
> The behavior i noticed was when the php started dieing, then all php
> processes would be in "pfault" or "sbwait" state.
>
> If you need any more infos i will be happy to provide. i just want to
> know what may be the cause of this.
>
> Information:
> 1 - This is what nginx_status page shows when php is close to dieing.
>
> Active connections: 5018
> server accepts handled requests
> 11836 11836 28147
> Reading: 49 Writing: 2103 Waiting: 2866
>
> 2 - my php-fpm.conf
>
>
> <?xml version="1.0" ?>
> <configuration>
>
> All relative paths in this config are relative to php's
> install prefix
>
> <section name="global_options">
>
> Pid file
> <value name="pid_file">/usr/local/kphp/logs/php-
> fpm.pid</value>
>
> Error log file
> <value name="error_log">/usr/local/kphp/logs/php-
> fpm.log</value>
>
> Log level
> <value name="log_level">notice</value>
>
> When this amount of php processes exited with SIGSEGV
> or SIGBUS ...
> <value name="emergency_restart_threshold">10</value>
>
> ... in a less than this interval of time, a graceful
> restart will be initiated.
> Useful to work around accidental curruptions in
> accelerator's shared memory.
> <value name="emergency_restart_interval">1m</value>
>
> Time limit on waiting child's reaction on signals from
> master
> <value name="process_control_timeout">5s</value>
>
> Set to 'no' to debug fpm
> <value name="daemonize">yes</value>
>
> </section>
>
> <workers>
>
> <section name="pool">
>
> Name of pool. Used in logs and stats.
> <value name="name">nginx</value>
>
> Address to accept fastcgi requests on.
> Valid syntax is 'ip.ad.re.ss:port' or just
> 'port' or '/path/to/unix/socket'
> <value name="listen_address">127.0.0.1:8000 http://127.0.0.1:8000</
> value>
>
> <value name="listen_options">
>
> Set listen(2) backlog
> <value name="backlog">-1</value>
>
> Set permissions for unix socket, if
> one used.
> In Linux read/write permissions must
> be set in order to allow connections from web server.
> Many BSD-derrived systems allow
> connections regardless of permissions.
> <value name="owner"></value>
> <value name="group"></value>
> <value name="mode">0666</value>
> </value>
>
> Additional php.ini defines, specific to this
> pool of workers.
> <value name="php_defines">
> 
> 
> </value>
>
> Unix user of processes
> <value name="user">nobody</value>
>
> Unix group of processes
> <value name="group">nobody</value>
>
> Process manager settings
> <value name="pm">
>
> Sets style of controling worker
> process count.
> Valid values are 'static' and 'apache-
> like'
> <value name="style">static</value>
>
> Sets the limit on the number of
> simultaneous requests that will be served.
> Equivalent to Apache MaxClients
> directive.
> Equivalent to PHP_FCGI_CHILDREN
> environment in original php.fcgi
> Used with any pm_style.
> <value name="max_children">256</value>
>
> Settings group for 'apache-like' pm
> style
> <value name="apache_like">
>
> Sets the number of server
> processes created on startup.
> Used only when 'apache-like'
> pm_style is selected
> <value name="StartServers">32</
> value>
>
> Sets the desired minimum
> number of idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MinSpareServers">64</value>
>
> Sets the desired maximum
> number of idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MaxSpareServers">128</value>
>
> </value>
>
> </value>
>
> The timeout (in seconds) for serving a single
> request after which the worker process will be terminated
> Should be used when 'max_execution_time' ini
> option does not stop script execution for some reason
> '0s' means 'off'
> <value name="request_terminate_timeout">0s</
> value>
>
> The timeout (in seconds) for serving of single
> request after which a php backtrace will be dumped to slow.log file
> '0s' means 'off'
> <value name="request_slowlog_timeout">10s</
> value>
>
> The log file for slow requests
> <value name="slowlog">logs/slow.log</value>
>
> Set open file desc rlimit
> <value name="rlimit_files">8092</value>
>
> Set max core size rlimit
> <value name="rlimit_core">0</value>
>
> Chroot to this directory at the start,
> absolute path
> <value name="chroot"></value>
>
> Chdir to this directory at the start, absolute
> path
> <value name="chdir"></value>
>
> Redirect workers' stdout and stderr into main
> error log.
> If not set, they will be redirected to /dev/
> null, according to FastCGI specs
> <value name="catch_workers_output">yes</value>
>
> How much requests each process should execute
> before respawn.
> Useful to work around memory leaks in 3rd
> party libraries.
> For endless request processing please specify
> 0
> Equivalent to PHP_FCGI_MAX_REQUESTS
> <value name="max_requests">500</value>
>
> Comma separated list of ipv4 addresses of
> FastCGI clients that allowed to connect.
> Equivalent to FCGI_WEB_SERVER_ADDRS
> environment in original php.fcgi (5.2.2+)
> Makes sense only with AF_INET listening
> socket.
> <value name="allowed_clients">127.0.0.1</
> value>
>
> Pass environment variables like
> LD_LIBRARY_PATH
> All $VARIABLEs are taken from current
> environment
> <value name="environment">
> <value name="HOSTNAME">$HOSTNAME</
> value>
> <value name="PATH">/usr/local/bin:/usr/
> bin:/bin</value>
> <value name="TMP">/tmp</value>
> <value name="TMPDIR">/tmp</value>
> <value name="TEMP">/tmp</value>
> <value name="OSTYPE">$OSTYPE</value>
> <value name="MACHTYPE">$MACHTYPE</
> value>
> <value name="MALLOC_CHECK_">2</value>
> </value>
>
> </section>
>
> </workers>
>
> </configuration>
>
> 3 - the relevant portion of nginx config:
>
> user nobody;
> worker_processes 64;
> worker_rlimit_nofile 10240;
>
> error_log logs/error.log;
> pid logs/nginx.pid;
>
> events {
> worker_connections 8192;
> use kqueue;
> }
>
> [...............................]
>
> location ~* \.php$ {
> fastcgi_pass 127.0.0.1:8000
> http://127.0.0.1:8000;
> fastcgi_index index.php;
> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
> public_html$fastcgi_script_name;
> fastcgi_param SCRIPT_NAME
> $fastcgi_script_name;
> fastcgi_param REQUEST_URI $request_uri;
> fastcgi_param DOCUMENT_URI $document_uri;
> fastcgi_param DOCUMENT_ROOT $document_root;
> fastcgi_param QUERY_STRING $query_string;
> fastcgi_param REQUEST_METHOD $request_method;
> fastcgi_param CONTENT_TYPE $content_type;
> fastcgi_param CONTENT_LENGTH $content_length;
> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
> fastcgi_param SERVER_SOFTWARE nginx;
> fastcgi_param REMOTE_ADDR $remote_addr;
> fastcgi_param REMOTE_PORT $remote_port;
> fastcgi_param SERVER_ADDR $server_addr;
> fastcgi_param SERVER_PORT $server_port;
> fastcgi_param SERVER_NAME $server_name;
> fastcgi_param SERVER_PROTOCOL
> $server_protocol;
>
> fastcgi_buffers 64 64k;
> fastcgi_buffer_size 64k;
>
>

Reply Quote

Jonathan Langevin

Re: Nginx high load - php stops at some point
September 28, 2010 01:32PM

What versions of nginx, php, and fpm are you running?
It appears to be older versions, due to the configurations that you have in
place.

Try setting (in FPM config):
catch_workers_output = yes
log_level = debug

I believe your version of FPM will have to have been compiled with debugging
enabled for that log level to function.
I'm also not certain catch_workers_output will function for you or not, but
I do know that since that setting has been enabled (which I changed at the
same time that I changed to APC), I have not seen the same errors again.

Be warned, debug log level is extremely verbose, expect logs to grow
exponentially.

-Jon

On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
themediaplanet90210@gmail.com> wrote:

> I tried APC prior to eAccelerator, and i had the same issues. It actually
> happened more often for php to hang than now with eaccelerator.
>
>
> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>
> I was experiencing similar issues recently.
> Changed my opcode cache to APC, haven't seen the same problem since.
>
> Try APC and see if you still have the same issues.
>
> -Jon
>
> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>
>> Hi guys,
>>
>> I am running a very busy website. It's running a photo gallery ->
>> Gallery2.
>>
>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>
>> At some point during the day, pretty much random, the load on the
>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>> cache.
>>
>> I have been debugging and i noticed that when the server reaches over
>> 6000 active connections, then everything gets clogged and eventually i
>> had to build a script that would restart php/nginx when the load
>> starts spiking, which is not quite a nice solution. I don't like doing
>> patchwork like this, but i had to.
>>
>> The behavior i noticed was when the php started dieing, then all php
>> processes would be in "pfault" or "sbwait" state.
>>
>> If you need any more infos i will be happy to provide. i just want to
>> know what may be the cause of this.
>>
>> Information:
>> 1 - This is what nginx_status page shows when php is close to dieing.
>>
>> Active connections: 5018
>> server accepts handled requests
>> 11836 11836 28147
>> Reading: 49 Writing: 2103 Waiting: 2866
>>
>> 2 - my php-fpm.conf
>>
>>
>> <?xml version="1.0" ?>
>> <configuration>
>>
>> All relative paths in this config are relative to php's
>> install prefix
>>
>> <section name="global_options">
>>
>> Pid file
>> <value name="pid_file">/usr/local/kphp/logs/php-
>> fpm.pid</value>
>>
>> Error log file
>> <value name="error_log">/usr/local/kphp/logs/php-
>> fpm.log</value>
>>
>> Log level
>> <value name="log_level">notice</value>
>>
>> When this amount of php processes exited with SIGSEGV
>> or SIGBUS ...
>> <value name="emergency_restart_threshold">10</value>
>>
>> ... in a less than this interval of time, a graceful
>> restart will be initiated.
>> Useful to work around accidental curruptions in
>> accelerator's shared memory.
>> <value name="emergency_restart_interval">1m</value>
>>
>> Time limit on waiting child's reaction on signals from
>> master
>> <value name="process_control_timeout">5s</value>
>>
>> Set to 'no' to debug fpm
>> <value name="daemonize">yes</value>
>>
>> </section>
>>
>> <workers>
>>
>> <section name="pool">
>>
>> Name of pool. Used in logs and stats.
>> <value name="name">nginx</value>
>>
>> Address to accept fastcgi requests on.
>> Valid syntax is 'ip.ad.re.ss:port' or just
>> 'port' or '/path/to/unix/socket'
>> <value name="listen_address">127.0.0.1:8000</
>> value>
>>
>> <value name="listen_options">
>>
>> Set listen(2) backlog
>> <value name="backlog">-1</value>
>>
>> Set permissions for unix socket, if
>> one used.
>> In Linux read/write permissions must
>> be set in order to allow connections from web server.
>> Many BSD-derrived systems allow
>> connections regardless of permissions.
>> <value name="owner"></value>
>> <value name="group"></value>
>> <value name="mode">0666</value>
>> </value>
>>
>> Additional php.ini defines, specific to this
>> pool of workers.
>> <value name="php_defines">
>> 
>> 
>> </value>
>>
>> Unix user of processes
>> <value name="user">nobody</value>
>>
>> Unix group of processes
>> <value name="group">nobody</value>
>>
>> Process manager settings
>> <value name="pm">
>>
>> Sets style of controling worker
>> process count.
>> Valid values are 'static' and 'apache-
>> like'
>> <value name="style">static</value>
>>
>> Sets the limit on the number of
>> simultaneous requests that will be served.
>> Equivalent to Apache MaxClients
>> directive.
>> Equivalent to PHP_FCGI_CHILDREN
>> environment in original php.fcgi
>> Used with any pm_style.
>> <value name="max_children">256</value>
>>
>> Settings group for 'apache-like' pm
>> style
>> <value name="apache_like">
>>
>> Sets the number of server
>> processes created on startup.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value name="StartServers">32</
>> value>
>>
>> Sets the desired minimum
>> number of idle server processes.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value
>> name="MinSpareServers">64</value>
>>
>> Sets the desired maximum
>> number of idle server processes.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value
>> name="MaxSpareServers">128</value>
>>
>> </value>
>>
>> </value>
>>
>> The timeout (in seconds) for serving a single
>> request after which the worker process will be terminated
>> Should be used when 'max_execution_time' ini
>> option does not stop script execution for some reason
>> '0s' means 'off'
>> <value name="request_terminate_timeout">0s</
>> value>
>>
>> The timeout (in seconds) for serving of single
>> request after which a php backtrace will be dumped to slow.log file
>> '0s' means 'off'
>> <value name="request_slowlog_timeout">10s</
>> value>
>>
>> The log file for slow requests
>> <value name="slowlog">logs/slow.log</value>
>>
>> Set open file desc rlimit
>> <value name="rlimit_files">8092</value>
>>
>> Set max core size rlimit
>> <value name="rlimit_core">0</value>
>>
>> Chroot to this directory at the start,
>> absolute path
>> <value name="chroot"></value>
>>
>> Chdir to this directory at the start, absolute
>> path
>> <value name="chdir"></value>
>>
>> Redirect workers' stdout and stderr into main
>> error log.
>> If not set, they will be redirected to /dev/
>> null, according to FastCGI specs
>> <value name="catch_workers_output">yes</value>
>>
>> How much requests each process should execute
>> before respawn.
>> Useful to work around memory leaks in 3rd
>> party libraries.
>> For endless request processing please specify
>> 0
>> Equivalent to PHP_FCGI_MAX_REQUESTS
>> <value name="max_requests">500</value>
>>
>> Comma separated list of ipv4 addresses of
>> FastCGI clients that allowed to connect.
>> Equivalent to FCGI_WEB_SERVER_ADDRS
>> environment in original php.fcgi (5.2.2+)
>> Makes sense only with AF_INET listening
>> socket.
>> <value name="allowed_clients">127.0.0.1</
>> value>
>>
>> Pass environment variables like
>> LD_LIBRARY_PATH
>> All $VARIABLEs are taken from current
>> environment
>> <value name="environment">
>> <value name="HOSTNAME">$HOSTNAME</
>> value>
>> <value name="PATH">/usr/local/bin:/usr/
>> bin:/bin</value>
>> <value name="TMP">/tmp</value>
>> <value name="TMPDIR">/tmp</value>
>> <value name="TEMP">/tmp</value>
>> <value name="OSTYPE">$OSTYPE</value>
>> <value name="MACHTYPE">$MACHTYPE</
>> value>
>> <value name="MALLOC_CHECK_">2</value>
>> </value>
>>
>> </section>
>>
>> </workers>
>>
>> </configuration>
>>
>> 3 - the relevant portion of nginx config:
>>
>> user nobody;
>> worker_processes 64;
>> worker_rlimit_nofile 10240;
>>
>> error_log logs/error.log;
>> pid logs/nginx.pid;
>>
>> events {
>> worker_connections 8192;
>> use kqueue;
>> }
>>
>> [...............................]
>>
>> location ~* \.php$ {
>> fastcgi_pass 127.0.0.1:8000;
>> fastcgi_index index.php;
>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>> public_html$fastcgi_script_name;
>> fastcgi_param SCRIPT_NAME
>> $fastcgi_script_name;
>> fastcgi_param REQUEST_URI $request_uri;
>> fastcgi_param DOCUMENT_URI $document_uri;
>> fastcgi_param DOCUMENT_ROOT $document_root;
>> fastcgi_param QUERY_STRING $query_string;
>> fastcgi_param REQUEST_METHOD $request_method;
>> fastcgi_param CONTENT_TYPE $content_type;
>> fastcgi_param CONTENT_LENGTH $content_length;
>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>> fastcgi_param SERVER_SOFTWARE nginx;
>> fastcgi_param REMOTE_ADDR $remote_addr;
>> fastcgi_param REMOTE_PORT $remote_port;
>> fastcgi_param SERVER_ADDR $server_addr;
>> fastcgi_param SERVER_PORT $server_port;
>> fastcgi_param SERVER_NAME $server_name;
>> fastcgi_param SERVER_PROTOCOL
>> $server_protocol;
>>
>> fastcgi_buffers 64 64k;
>> fastcgi_buffer_size 64k;
>>
>
>
>

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 28, 2010 01:38PM

I already specified the versions of everything in the first post. php
5.2.10 / nginx latest - 0.8.52.

I am going to check the debug options about now.

On 9/28/2010 7:31 PM, Jonathan Langevin wrote:
> What versions of nginx, php, and fpm are you running?
> It appears to be older versions, due to the configurations that you
> have in place.
>
> Try setting (in FPM config):
> catch_workers_output = yes
> log_level = debug
>
> I believe your version of FPM will have to have been compiled with
> debugging enabled for that log level to function.
> I'm also not certain catch_workers_output will function for you or
> not, but I do know that since that setting has been enabled (which I
> changed at the same time that I changed to APC), I have not seen the
> same errors again.
>
> Be warned, debug log level is extremely verbose, expect logs to grow
> exponentially.
>
> -Jon
>
> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu
> <themediaplanet90210@gmail.com <mailto:themediaplanet90210@gmail.com>>
> wrote:
>
> I tried APC prior to eAccelerator, and i had the same issues. It
> actually happened more often for php to hang than now with
> eaccelerator.
>
>
> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>> I was experiencing similar issues recently.
>> Changed my opcode cache to APC, haven't seen the same problem since.
>>
>> Try APC and see if you still have the same issues.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:15 PM, katmai
>> <themediaplanet90210@gmail.com
>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>
>> Hi guys,
>>
>> I am running a very busy website. It's running a photo gallery ->
>> Gallery2.
>>
>> The system is a Quad core Xeon, with 16 gb RAM, and separate
>> drives
>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>
>> At some point during the day, pretty much random, the load on the
>> server spikes, and then php processing stops. I run nginx
>> 0.8.52, and
>> php 5.2.10 with fpm patch, and i am using eAccelerator as an
>> opcode
>> cache.
>>
>> I have been debugging and i noticed that when the server
>> reaches over
>> 6000 active connections, then everything gets clogged and
>> eventually i
>> had to build a script that would restart php/nginx when the load
>> starts spiking, which is not quite a nice solution. I don't
>> like doing
>> patchwork like this, but i had to.
>>
>> The behavior i noticed was when the php started dieing, then
>> all php
>> processes would be in "pfault" or "sbwait" state.
>>
>> If you need any more infos i will be happy to provide. i just
>> want to
>> know what may be the cause of this.
>>
>> Information:
>> 1 - This is what nginx_status page shows when php is close to
>> dieing.
>>
>> Active connections: 5018
>> server accepts handled requests
>> 11836 11836 28147
>> Reading: 49 Writing: 2103 Waiting: 2866
>>
>> 2 - my php-fpm.conf
>>
>>
>> <?xml version="1.0" ?>
>> <configuration>
>>
>> All relative paths in this config are relative to php's
>> install prefix
>>
>> <section name="global_options">
>>
>> Pid file
>> <value name="pid_file">/usr/local/kphp/logs/php-
>> fpm.pid</value>
>>
>> Error log file
>> <value name="error_log">/usr/local/kphp/logs/php-
>> fpm.log</value>
>>
>> Log level
>> <value name="log_level">notice</value>
>>
>> When this amount of php processes exited with
>> SIGSEGV
>> or SIGBUS ...
>> <value name="emergency_restart_threshold">10</value>
>>
>> ... in a less than this interval of time, a
>> graceful
>> restart will be initiated.
>> Useful to work around accidental curruptions in
>> accelerator's shared memory.
>> <value name="emergency_restart_interval">1m</value>
>>
>> Time limit on waiting child's reaction on
>> signals from
>> master
>> <value name="process_control_timeout">5s</value>
>>
>> Set to 'no' to debug fpm
>> <value name="daemonize">yes</value>
>>
>> </section>
>>
>> <workers>
>>
>> <section name="pool">
>>
>> Name of pool. Used in logs and stats.
>> <value name="name">nginx</value>
>>
>> Address to accept fastcgi requests on.
>> Valid syntax is 'ip.ad.re.ss:port' or just
>> 'port' or '/path/to/unix/socket'
>> <value name="listen_address">127.0.0.1:8000
>> http://127.0.0.1:8000</
>> value>
>>
>> <value name="listen_options">
>>
>> Set listen(2) backlog
>> <value name="backlog">-1</value>
>>
>> Set permissions for unix
>> socket, if
>> one used.
>> In Linux read/write
>> permissions must
>> be set in order to allow connections from web server.
>> Many BSD-derrived systems allow
>> connections regardless of permissions.
>> <value name="owner"></value>
>> <value name="group"></value>
>> <value name="mode">0666</value>
>> </value>
>>
>> Additional php.ini defines, specific
>> to this
>> pool of workers.
>> <value name="php_defines">
>> 
>> 
>> </value>
>>
>> Unix user of processes
>> <value name="user">nobody</value>
>>
>> Unix group of processes
>> <value name="group">nobody</value>
>>
>> Process manager settings
>> <value name="pm">
>>
>> Sets style of controling worker
>> process count.
>> Valid values are 'static' and
>> 'apache-
>> like'
>> <value name="style">static</value>
>>
>> Sets the limit on the number of
>> simultaneous requests that will be served.
>> Equivalent to Apache MaxClients
>> directive.
>> Equivalent to PHP_FCGI_CHILDREN
>> environment in original php.fcgi
>> Used with any pm_style.
>> <value name="max_children">256</value>
>>
>> Settings group for
>> 'apache-like' pm
>> style
>> <value name="apache_like">
>>
>> Sets the number of server
>> processes created on startup.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value name="StartServers">32</
>> value>
>>
>> Sets the desired minimum
>> number of idle server processes.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value
>> name="MinSpareServers">64</value>
>>
>> Sets the desired maximum
>> number of idle server processes.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value
>> name="MaxSpareServers">128</value>
>>
>> </value>
>>
>> </value>
>>
>> The timeout (in seconds) for serving a
>> single
>> request after which the worker process will be terminated
>> Should be used when
>> 'max_execution_time' ini
>> option does not stop script execution for some reason
>> '0s' means 'off'
>> <value name="request_terminate_timeout">0s</
>> value>
>>
>> The timeout (in seconds) for serving
>> of single
>> request after which a php backtrace will be dumped to
>> slow.log file
>> '0s' means 'off'
>> <value name="request_slowlog_timeout">10s</
>> value>
>>
>> The log file for slow requests
>> <value name="slowlog">logs/slow.log</value>
>>
>> Set open file desc rlimit
>> <value name="rlimit_files">8092</value>
>>
>> Set max core size rlimit
>> <value name="rlimit_core">0</value>
>>
>> Chroot to this directory at the start,
>> absolute path
>> <value name="chroot"></value>
>>
>> Chdir to this directory at the start,
>> absolute
>> path
>> <value name="chdir"></value>
>>
>> Redirect workers' stdout and stderr
>> into main
>> error log.
>> If not set, they will be redirected to
>> /dev/
>> null, according to FastCGI specs
>> <value name="catch_workers_output">yes</value>
>>
>> How much requests each process should
>> execute
>> before respawn.
>> Useful to work around memory leaks in 3rd
>> party libraries.
>> For endless request processing please
>> specify
>> 0
>> Equivalent to PHP_FCGI_MAX_REQUESTS
>> <value name="max_requests">500</value>
>>
>> Comma separated list of ipv4 addresses of
>> FastCGI clients that allowed to connect.
>> Equivalent to FCGI_WEB_SERVER_ADDRS
>> environment in original php.fcgi (5.2.2+)
>> Makes sense only with AF_INET listening
>> socket.
>> <value name="allowed_clients">127.0.0.1</
>> value>
>>
>> Pass environment variables like
>> LD_LIBRARY_PATH
>> All $VARIABLEs are taken from current
>> environment
>> <value name="environment">
>> <value name="HOSTNAME">$HOSTNAME</
>> value>
>> <value name="PATH">/usr/local/bin:/usr/
>> bin:/bin</value>
>> <value name="TMP">/tmp</value>
>> <value name="TMPDIR">/tmp</value>
>> <value name="TEMP">/tmp</value>
>> <value name="OSTYPE">$OSTYPE</value>
>> <value name="MACHTYPE">$MACHTYPE</
>> value>
>> <value name="MALLOC_CHECK_">2</value>
>> </value>
>>
>> </section>
>>
>> </workers>
>>
>> </configuration>
>>
>> 3 - the relevant portion of nginx config:
>>
>> user nobody;
>> worker_processes 64;
>> worker_rlimit_nofile 10240;
>>
>> error_log logs/error.log;
>> pid logs/nginx.pid;
>>
>> events {
>> worker_connections 8192;
>> use kqueue;
>> }
>>
>> [...............................]
>>
>> location ~* \.php$ {
>> fastcgi_pass 127.0.0.1:8000
>> http://127.0.0.1:8000;
>> fastcgi_index index.php;
>> fastcgi_param SCRIPT_FILENAME
>> /xdrive/cityv3/
>> public_html$fastcgi_script_name;
>> fastcgi_param SCRIPT_NAME
>> $fastcgi_script_name;
>> fastcgi_param REQUEST_URI
>> $request_uri;
>> fastcgi_param DOCUMENT_URI
>> $document_uri;
>> fastcgi_param DOCUMENT_ROOT
>> $document_root;
>> fastcgi_param QUERY_STRING
>> $query_string;
>> fastcgi_param REQUEST_METHOD
>> $request_method;
>> fastcgi_param CONTENT_TYPE
>> $content_type;
>> fastcgi_param CONTENT_LENGTH
>> $content_length;
>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>> fastcgi_param SERVER_SOFTWARE nginx;
>> fastcgi_param REMOTE_ADDR
>> $remote_addr;
>> fastcgi_param REMOTE_PORT
>> $remote_port;
>> fastcgi_param SERVER_ADDR
>> $server_addr;
>> fastcgi_param SERVER_PORT
>> $server_port;
>> fastcgi_param SERVER_NAME
>> $server_name;
>> fastcgi_param SERVER_PROTOCOL
>> $server_protocol;
>>
>> fastcgi_buffers 64 64k;
>> fastcgi_buffer_size 64k;
>>
>>
>
>

Reply Quote

Jonathan Langevin

Re: Nginx high load - php stops at some point
September 28, 2010 01:40PM

My apologies on versions, I missed that in my follow-up.

On Tue, Sep 28, 2010 at 1:37 PM, StefanitaRares Dumitrescu <
themediaplanet90210@gmail.com> wrote:

> I already specified the versions of everything in the first post. php
> 5.2.10 / nginx latest - 0.8.52.
>
> I am going to check the debug options about now.
>
> On 9/28/2010 7:31 PM, Jonathan Langevin wrote:
>
> What versions of nginx, php, and fpm are you running?
> It appears to be older versions, due to the configurations that you have in
> place.
>
> Try setting (in FPM config):
> catch_workers_output = yes
> log_level = debug
>
> I believe your version of FPM will have to have been compiled with
> debugging enabled for that log level to function.
> I'm also not certain catch_workers_output will function for you or not, but
> I do know that since that setting has been enabled (which I changed at the
> same time that I changed to APC), I have not seen the same errors again.
>
> Be warned, debug log level is extremely verbose, expect logs to grow
> exponentially.
>
> -Jon
>
> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
> themediaplanet90210@gmail.com> wrote:
>
>> I tried APC prior to eAccelerator, and i had the same issues. It actually
>> happened more often for php to hang than now with eaccelerator.
>>
>>
>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>
>> I was experiencing similar issues recently.
>> Changed my opcode cache to APC, haven't seen the same problem since.
>>
>> Try APC and see if you still have the same issues.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>>
>>> Hi guys,
>>>
>>> I am running a very busy website. It's running a photo gallery ->
>>> Gallery2.
>>>
>>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>>
>>> At some point during the day, pretty much random, the load on the
>>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>>> cache.
>>>
>>> I have been debugging and i noticed that when the server reaches over
>>> 6000 active connections, then everything gets clogged and eventually i
>>> had to build a script that would restart php/nginx when the load
>>> starts spiking, which is not quite a nice solution. I don't like doing
>>> patchwork like this, but i had to.
>>>
>>> The behavior i noticed was when the php started dieing, then all php
>>> processes would be in "pfault" or "sbwait" state.
>>>
>>> If you need any more infos i will be happy to provide. i just want to
>>> know what may be the cause of this.
>>>
>>> Information:
>>> 1 - This is what nginx_status page shows when php is close to dieing.
>>>
>>> Active connections: 5018
>>> server accepts handled requests
>>> 11836 11836 28147
>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>
>>> 2 - my php-fpm.conf
>>>
>>>
>>> <?xml version="1.0" ?>
>>> <configuration>
>>>
>>> All relative paths in this config are relative to php's
>>> install prefix
>>>
>>> <section name="global_options">
>>>
>>> Pid file
>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>> fpm.pid</value>
>>>
>>> Error log file
>>> <value name="error_log">/usr/local/kphp/logs/php-
>>> fpm.log</value>
>>>
>>> Log level
>>> <value name="log_level">notice</value>
>>>
>>> When this amount of php processes exited with SIGSEGV
>>> or SIGBUS ...
>>> <value name="emergency_restart_threshold">10</value>
>>>
>>> ... in a less than this interval of time, a graceful
>>> restart will be initiated.
>>> Useful to work around accidental curruptions in
>>> accelerator's shared memory.
>>> <value name="emergency_restart_interval">1m</value>
>>>
>>> Time limit on waiting child's reaction on signals from
>>> master
>>> <value name="process_control_timeout">5s</value>
>>>
>>> Set to 'no' to debug fpm
>>> <value name="daemonize">yes</value>
>>>
>>> </section>
>>>
>>> <workers>
>>>
>>> <section name="pool">
>>>
>>> Name of pool. Used in logs and stats.
>>> <value name="name">nginx</value>
>>>
>>> Address to accept fastcgi requests on.
>>> Valid syntax is 'ip.ad.re.ss:port' or just
>>> 'port' or '/path/to/unix/socket'
>>> <value name="listen_address">127.0.0.1:8000</
>>> value>
>>>
>>> <value name="listen_options">
>>>
>>> Set listen(2) backlog
>>> <value name="backlog">-1</value>
>>>
>>> Set permissions for unix socket, if
>>> one used.
>>> In Linux read/write permissions must
>>> be set in order to allow connections from web server.
>>> Many BSD-derrived systems allow
>>> connections regardless of permissions.
>>> <value name="owner"></value>
>>> <value name="group"></value>
>>> <value name="mode">0666</value>
>>> </value>
>>>
>>> Additional php.ini defines, specific to this
>>> pool of workers.
>>> <value name="php_defines">
>>> 
>>> 
>>> </value>
>>>
>>> Unix user of processes
>>> <value name="user">nobody</value>
>>>
>>> Unix group of processes
>>> <value name="group">nobody</value>
>>>
>>> Process manager settings
>>> <value name="pm">
>>>
>>> Sets style of controling worker
>>> process count.
>>> Valid values are 'static' and 'apache-
>>> like'
>>> <value name="style">static</value>
>>>
>>> Sets the limit on the number of
>>> simultaneous requests that will be served.
>>> Equivalent to Apache MaxClients
>>> directive.
>>> Equivalent to PHP_FCGI_CHILDREN
>>> environment in original php.fcgi
>>> Used with any pm_style.
>>> <value name="max_children">256</value>
>>>
>>> Settings group for 'apache-like' pm
>>> style
>>> <value name="apache_like">
>>>
>>> Sets the number of server
>>> processes created on startup.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value name="StartServers">32</
>>> value>
>>>
>>> Sets the desired minimum
>>> number of idle server processes.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MinSpareServers">64</value>
>>>
>>> Sets the desired maximum
>>> number of idle server processes.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MaxSpareServers">128</value>
>>>
>>> </value>
>>>
>>> </value>
>>>
>>> The timeout (in seconds) for serving a single
>>> request after which the worker process will be terminated
>>> Should be used when 'max_execution_time' ini
>>> option does not stop script execution for some reason
>>> '0s' means 'off'
>>> <value name="request_terminate_timeout">0s</
>>> value>
>>>
>>> The timeout (in seconds) for serving of single
>>> request after which a php backtrace will be dumped to slow.log file
>>> '0s' means 'off'
>>> <value name="request_slowlog_timeout">10s</
>>> value>
>>>
>>> The log file for slow requests
>>> <value name="slowlog">logs/slow.log</value>
>>>
>>> Set open file desc rlimit
>>> <value name="rlimit_files">8092</value>
>>>
>>> Set max core size rlimit
>>> <value name="rlimit_core">0</value>
>>>
>>> Chroot to this directory at the start,
>>> absolute path
>>> <value name="chroot"></value>
>>>
>>> Chdir to this directory at the start, absolute
>>> path
>>> <value name="chdir"></value>
>>>
>>> Redirect workers' stdout and stderr into main
>>> error log.
>>> If not set, they will be redirected to /dev/
>>> null, according to FastCGI specs
>>> <value name="catch_workers_output">yes</value>
>>>
>>> How much requests each process should execute
>>> before respawn.
>>> Useful to work around memory leaks in 3rd
>>> party libraries.
>>> For endless request processing please specify
>>> 0
>>> Equivalent to PHP_FCGI_MAX_REQUESTS
>>> <value name="max_requests">500</value>
>>>
>>> Comma separated list of ipv4 addresses of
>>> FastCGI clients that allowed to connect.
>>> Equivalent to FCGI_WEB_SERVER_ADDRS
>>> environment in original php.fcgi (5.2.2+)
>>> Makes sense only with AF_INET listening
>>> socket.
>>> <value name="allowed_clients">127.0.0.1</
>>> value>
>>>
>>> Pass environment variables like
>>> LD_LIBRARY_PATH
>>> All $VARIABLEs are taken from current
>>> environment
>>> <value name="environment">
>>> <value name="HOSTNAME">$HOSTNAME</
>>> value>
>>> <value name="PATH">/usr/local/bin:/usr/
>>> bin:/bin</value>
>>> <value name="TMP">/tmp</value>
>>> <value name="TMPDIR">/tmp</value>
>>> <value name="TEMP">/tmp</value>
>>> <value name="OSTYPE">$OSTYPE</value>
>>> <value name="MACHTYPE">$MACHTYPE</
>>> value>
>>> <value name="MALLOC_CHECK_">2</value>
>>> </value>
>>>
>>> </section>
>>>
>>> </workers>
>>>
>>> </configuration>
>>>
>>> 3 - the relevant portion of nginx config:
>>>
>>> user nobody;
>>> worker_processes 64;
>>> worker_rlimit_nofile 10240;
>>>
>>> error_log logs/error.log;
>>> pid logs/nginx.pid;
>>>
>>> events {
>>> worker_connections 8192;
>>> use kqueue;
>>> }
>>>
>>> [...............................]
>>>
>>> location ~* \.php$ {
>>> fastcgi_pass 127.0.0.1:8000;
>>> fastcgi_index index.php;
>>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>>> public_html$fastcgi_script_name;
>>> fastcgi_param SCRIPT_NAME
>>> $fastcgi_script_name;
>>> fastcgi_param REQUEST_URI $request_uri;
>>> fastcgi_param DOCUMENT_URI $document_uri;
>>> fastcgi_param DOCUMENT_ROOT $document_root;
>>> fastcgi_param QUERY_STRING $query_string;
>>> fastcgi_param REQUEST_METHOD $request_method;
>>> fastcgi_param CONTENT_TYPE $content_type;
>>> fastcgi_param CONTENT_LENGTH $content_length;
>>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>>> fastcgi_param SERVER_SOFTWARE nginx;
>>> fastcgi_param REMOTE_ADDR $remote_addr;
>>> fastcgi_param REMOTE_PORT $remote_port;
>>> fastcgi_param SERVER_ADDR $server_addr;
>>> fastcgi_param SERVER_PORT $server_port;
>>> fastcgi_param SERVER_NAME $server_name;
>>> fastcgi_param SERVER_PROTOCOL
>>> $server_protocol;
>>>
>>> fastcgi_buffers 64 64k;
>>> fastcgi_buffer_size 64k;
>>>
>>
>>
>>
>
>

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 28, 2010 02:18PM

I upgraded to the latest php 5.2 - 5.2.14 with php-fpm 0.5.14, just to
take the "old version thing" out of equation.

I have made a few small modifications, to the php-fpm.conf, i increased
the number of minservers and max servers and start servers. The traffic
is lower now so i cannot really know for sure this is works, as the php
crash/hang usually happens during the day.

I also decreased the number of nginx workers to 8 instead of 64, as i
think it made no sense to have 64, considering i have 4 cores with HT.

Used with any pm_style.
<value name="max_children">256</value>
Settings group for 'apache-like' pm style
<value name="apache_like">
Sets the number of server
processes created on startup.
Used only when 'apache-like'
pm_style is selected
<value name="StartServers">64</value>
Sets the desired minimum number
of idle server processes.
Used only when 'apache-like'
pm_style is selected
<value name="MinSpareServers">128</value>
Sets the desired maximum number
of idle server processes.
Used only when 'apache-like'
pm_style is selected
<value name="MaxSpareServers">256</value>
</value>
</value>

On 9/28/2010 7:38 PM, Jonathan Langevin wrote:
> My apologies on versions, I missed that in my follow-up.
>
> On Tue, Sep 28, 2010 at 1:37 PM, StefanitaRares Dumitrescu
> <themediaplanet90210@gmail.com <mailto:themediaplanet90210@gmail.com>>
> wrote:
>
> I already specified the versions of everything in the first post.
> php 5.2.10 / nginx latest - 0.8.52.
>
> I am going to check the debug options about now.
>
> On 9/28/2010 7:31 PM, Jonathan Langevin wrote:
>> What versions of nginx, php, and fpm are you running?
>> It appears to be older versions, due to the configurations that
>> you have in place.
>>
>> Try setting (in FPM config):
>> catch_workers_output = yes
>> log_level = debug
>>
>> I believe your version of FPM will have to have been compiled
>> with debugging enabled for that log level to function.
>> I'm also not certain catch_workers_output will function for you
>> or not, but I do know that since that setting has been enabled
>> (which I changed at the same time that I changed to APC), I have
>> not seen the same errors again.
>>
>> Be warned, debug log level is extremely verbose, expect logs to
>> grow exponentially.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu
>> <themediaplanet90210@gmail.com
>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>
>> I tried APC prior to eAccelerator, and i had the same issues.
>> It actually happened more often for php to hang than now with
>> eaccelerator.
>>
>>
>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>> I was experiencing similar issues recently.
>>> Changed my opcode cache to APC, haven't seen the same
>>> problem since.
>>>
>>> Try APC and see if you still have the same issues.
>>>
>>> -Jon
>>>
>>> On Tue, Sep 28, 2010 at 1:15 PM, katmai
>>> <themediaplanet90210@gmail.com
>>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>>
>>> Hi guys,
>>>
>>> I am running a very busy website. It's running a photo
>>> gallery ->
>>> Gallery2.
>>>
>>> The system is a Quad core Xeon, with 16 gb RAM, and
>>> separate drives
>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>>
>>> At some point during the day, pretty much random, the
>>> load on the
>>> server spikes, and then php processing stops. I run
>>> nginx 0.8.52, and
>>> php 5.2.10 with fpm patch, and i am using eAccelerator
>>> as an opcode
>>> cache.
>>>
>>> I have been debugging and i noticed that when the server
>>> reaches over
>>> 6000 active connections, then everything gets clogged
>>> and eventually i
>>> had to build a script that would restart php/nginx when
>>> the load
>>> starts spiking, which is not quite a nice solution. I
>>> don't like doing
>>> patchwork like this, but i had to.
>>>
>>> The behavior i noticed was when the php started dieing,
>>> then all php
>>> processes would be in "pfault" or "sbwait" state.
>>>
>>> If you need any more infos i will be happy to provide. i
>>> just want to
>>> know what may be the cause of this.
>>>
>>> Information:
>>> 1 - This is what nginx_status page shows when php is
>>> close to dieing.
>>>
>>> Active connections: 5018
>>> server accepts handled requests
>>> 11836 11836 28147
>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>
>>> 2 - my php-fpm.conf
>>>
>>>
>>> <?xml version="1.0" ?>
>>> <configuration>
>>>
>>> All relative paths in this config are relative to
>>> php's
>>> install prefix
>>>
>>> <section name="global_options">
>>>
>>> Pid file
>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>> fpm.pid</value>
>>>
>>> Error log file
>>> <value name="error_log">/usr/local/kphp/logs/php-
>>> fpm.log</value>
>>>
>>> Log level
>>> <value name="log_level">notice</value>
>>>
>>> When this amount of php processes exited
>>> with SIGSEGV
>>> or SIGBUS ...
>>> <value name="emergency_restart_threshold">10</value>
>>>
>>> ... in a less than this interval of time,
>>> a graceful
>>> restart will be initiated.
>>> Useful to work around accidental
>>> curruptions in
>>> accelerator's shared memory.
>>> <value name="emergency_restart_interval">1m</value>
>>>
>>> Time limit on waiting child's reaction on
>>> signals from
>>> master
>>> <value name="process_control_timeout">5s</value>
>>>
>>> Set to 'no' to debug fpm
>>> <value name="daemonize">yes</value>
>>>
>>> </section>
>>>
>>> <workers>
>>>
>>> <section name="pool">
>>>
>>> Name of pool. Used in logs and stats.
>>> <value name="name">nginx</value>
>>>
>>> Address to accept fastcgi
>>> requests on.
>>> Valid syntax is
>>> 'ip.ad.re.ss:port' or just
>>> 'port' or '/path/to/unix/socket'
>>> <value name="listen_address">127.0.0.1:8000
>>> http://127.0.0.1:8000</
>>> value>
>>>
>>> <value name="listen_options">
>>>
>>> Set listen(2) backlog
>>> <value name="backlog">-1</value>
>>>
>>> Set permissions for unix
>>> socket, if
>>> one used.
>>> In Linux read/write
>>> permissions must
>>> be set in order to allow connections from web server.
>>> Many BSD-derrived systems
>>> allow
>>> connections regardless of permissions.
>>> <value name="owner"></value>
>>> <value name="group"></value>
>>> <value name="mode">0666</value>
>>> </value>
>>>
>>> Additional php.ini defines,
>>> specific to this
>>> pool of workers.
>>> <value name="php_defines">
>>> 
>>> 
>>> </value>
>>>
>>> Unix user of processes
>>> <value name="user">nobody</value>
>>>
>>> Unix group of processes
>>> <value name="group">nobody</value>
>>>
>>> Process manager settings
>>> <value name="pm">
>>>
>>> Sets style of controling
>>> worker
>>> process count.
>>> Valid values are 'static'
>>> and 'apache-
>>> like'
>>> <value name="style">static</value>
>>>
>>> Sets the limit on the
>>> number of
>>> simultaneous requests that will be served.
>>> Equivalent to Apache
>>> MaxClients
>>> directive.
>>> Equivalent to
>>> PHP_FCGI_CHILDREN
>>> environment in original php.fcgi
>>> Used with any pm_style.
>>> <value name="max_children">256</value>
>>>
>>> Settings group for
>>> 'apache-like' pm
>>> style
>>> <value name="apache_like">
>>>
>>> Sets the number
>>> of server
>>> processes created on startup.
>>> Used only when
>>> 'apache-like'
>>> pm_style is selected
>>> <value name="StartServers">32</
>>> value>
>>>
>>> Sets the desired
>>> minimum
>>> number of idle server processes.
>>> Used only when
>>> 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MinSpareServers">64</value>
>>>
>>> Sets the desired
>>> maximum
>>> number of idle server processes.
>>> Used only when
>>> 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MaxSpareServers">128</value>
>>>
>>> </value>
>>>
>>> </value>
>>>
>>> The timeout (in seconds) for
>>> serving a single
>>> request after which the worker process will be terminated
>>> Should be used when
>>> 'max_execution_time' ini
>>> option does not stop script execution for some reason
>>> '0s' means 'off'
>>> <value name="request_terminate_timeout">0s</
>>> value>
>>>
>>> The timeout (in seconds) for
>>> serving of single
>>> request after which a php backtrace will be dumped to
>>> slow.log file
>>> '0s' means 'off'
>>> <value name="request_slowlog_timeout">10s</
>>> value>
>>>
>>> The log file for slow requests
>>> <value name="slowlog">logs/slow.log</value>
>>>
>>> Set open file desc rlimit
>>> <value name="rlimit_files">8092</value>
>>>
>>> Set max core size rlimit
>>> <value name="rlimit_core">0</value>
>>>
>>> Chroot to this directory at the
>>> start,
>>> absolute path
>>> <value name="chroot"></value>
>>>
>>> Chdir to this directory at the
>>> start, absolute
>>> path
>>> <value name="chdir"></value>
>>>
>>> Redirect workers' stdout and
>>> stderr into main
>>> error log.
>>> If not set, they will be
>>> redirected to /dev/
>>> null, according to FastCGI specs
>>> <value name="catch_workers_output">yes</value>
>>>
>>> How much requests each process
>>> should execute
>>> before respawn.
>>> Useful to work around memory
>>> leaks in 3rd
>>> party libraries.
>>> For endless request processing
>>> please specify
>>> 0
>>> Equivalent to PHP_FCGI_MAX_REQUESTS
>>> <value name="max_requests">500</value>
>>>
>>> Comma separated list of ipv4
>>> addresses of
>>> FastCGI clients that allowed to connect.
>>> Equivalent to FCGI_WEB_SERVER_ADDRS
>>> environment in original php.fcgi (5.2.2+)
>>> Makes sense only with AF_INET
>>> listening
>>> socket.
>>> <value name="allowed_clients">127.0.0.1</
>>> value>
>>>
>>> Pass environment variables like
>>> LD_LIBRARY_PATH
>>> All $VARIABLEs are taken from current
>>> environment
>>> <value name="environment">
>>> <value name="HOSTNAME">$HOSTNAME</
>>> value>
>>> <value name="PATH">/usr/local/bin:/usr/
>>> bin:/bin</value>
>>> <value name="TMP">/tmp</value>
>>> <value name="TMPDIR">/tmp</value>
>>> <value name="TEMP">/tmp</value>
>>> <value name="OSTYPE">$OSTYPE</value>
>>> <value name="MACHTYPE">$MACHTYPE</
>>> value>
>>> <value name="MALLOC_CHECK_">2</value>
>>> </value>
>>>
>>> </section>
>>>
>>> </workers>
>>>
>>> </configuration>
>>>
>>> 3 - the relevant portion of nginx config:
>>>
>>> user nobody;
>>> worker_processes 64;
>>> worker_rlimit_nofile 10240;
>>>
>>> error_log logs/error.log;
>>> pid logs/nginx.pid;
>>>
>>> events {
>>> worker_connections 8192;
>>> use kqueue;
>>> }
>>>
>>> [...............................]
>>>
>>> location ~* \.php$ {
>>> fastcgi_pass 127.0.0.1:8000
>>> http://127.0.0.1:8000;
>>> fastcgi_index index.php;
>>> fastcgi_param SCRIPT_FILENAME
>>> /xdrive/cityv3/
>>> public_html$fastcgi_script_name;
>>> fastcgi_param SCRIPT_NAME
>>> $fastcgi_script_name;
>>> fastcgi_param REQUEST_URI
>>> $request_uri;
>>> fastcgi_param DOCUMENT_URI
>>> $document_uri;
>>> fastcgi_param DOCUMENT_ROOT
>>> $document_root;
>>> fastcgi_param QUERY_STRING
>>> $query_string;
>>> fastcgi_param REQUEST_METHOD
>>> $request_method;
>>> fastcgi_param CONTENT_TYPE
>>> $content_type;
>>> fastcgi_param CONTENT_LENGTH
>>> $content_length;
>>> fastcgi_param GATEWAY_INTERFACE
>>> CGI/1.1;
>>> fastcgi_param SERVER_SOFTWARE nginx;
>>> fastcgi_param REMOTE_ADDR
>>> $remote_addr;
>>> fastcgi_param REMOTE_PORT
>>> $remote_port;
>>> fastcgi_param SERVER_ADDR
>>> $server_addr;
>>> fastcgi_param SERVER_PORT
>>> $server_port;
>>> fastcgi_param SERVER_NAME
>>> $server_name;
>>> fastcgi_param SERVER_PROTOCOL
>>> $server_protocol;
>>>
>>> fastcgi_buffers 64 64k;
>>> fastcgi_buffer_size 64k;
>>>
>>>
>>
>>
>
>

Reply Quote

Jonathan Langevin

Re: Nginx high load - php stops at some point
September 28, 2010 02:26PM

You may be able to replicate the problem by running siege against the
webserver.
I typically set up a known heavy PHP page (you can also provide a list of
urls, accessed randomly) to be the target. It's good at least for getting an
idea of how many concurrent connections you can handle.

http://www.joedog.org/index/siege-home

On Tue, Sep 28, 2010 at 2:16 PM, StefanitaRares Dumitrescu <
themediaplanet90210@gmail.com> wrote:

> I upgraded to the latest php 5.2 - 5.2.14 with php-fpm 0.5.14, just to
> take the "old version thing" out of equation.
>
> I have made a few small modifications, to the php-fpm.conf, i increased
> the number of minservers and max servers and start servers. The traffic is
> lower now so i cannot really know for sure this is works, as the php
> crash/hang usually happens during the day.
>
> I also decreased the number of nginx workers to 8 instead of 64, as i think
> it made no sense to have 64, considering i have 4 cores with HT.
>
>
> Used with any pm_style.
> <value name="max_children">256</value>
> Settings group for 'apache-like' pm style
> <value name="apache_like">
> Sets the number of server processes
> created on startup.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="StartServers">64</value>
>
> Sets the desired minimum number of
> idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MinSpareServers">128</value>
>
> Sets the desired maximum number of
> idle server processes.
> Used only when 'apache-like'
> pm_style is selected
> <value
> name="MaxSpareServers">256</value>
> </value>
> </value>
>
>
>
> On 9/28/2010 7:38 PM, Jonathan Langevin wrote:
>
> My apologies on versions, I missed that in my follow-up.
>
> On Tue, Sep 28, 2010 at 1:37 PM, StefanitaRares Dumitrescu <
> themediaplanet90210@gmail.com> wrote:
>
>> I already specified the versions of everything in the first post. php
>> 5.2.10 / nginx latest - 0.8.52.
>>
>> I am going to check the debug options about now.
>>
>> On 9/28/2010 7:31 PM, Jonathan Langevin wrote:
>>
>> What versions of nginx, php, and fpm are you running?
>> It appears to be older versions, due to the configurations that you have
>> in place.
>>
>> Try setting (in FPM config):
>> catch_workers_output = yes
>> log_level = debug
>>
>> I believe your version of FPM will have to have been compiled with
>> debugging enabled for that log level to function.
>> I'm also not certain catch_workers_output will function for you or not,
>> but I do know that since that setting has been enabled (which I changed at
>> the same time that I changed to APC), I have not seen the same errors again.
>>
>> Be warned, debug log level is extremely verbose, expect logs to grow
>> exponentially.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
>> themediaplanet90210@gmail.com> wrote:
>>
>>> I tried APC prior to eAccelerator, and i had the same issues. It
>>> actually happened more often for php to hang than now with eaccelerator.
>>>
>>>
>>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>>
>>> I was experiencing similar issues recently.
>>> Changed my opcode cache to APC, haven't seen the same problem since.
>>>
>>> Try APC and see if you still have the same issues.
>>>
>>> -Jon
>>>
>>> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I am running a very busy website. It's running a photo gallery ->
>>>> Gallery2.
>>>>
>>>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>>>
>>>> At some point during the day, pretty much random, the load on the
>>>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>>>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>>>> cache.
>>>>
>>>> I have been debugging and i noticed that when the server reaches over
>>>> 6000 active connections, then everything gets clogged and eventually i
>>>> had to build a script that would restart php/nginx when the load
>>>> starts spiking, which is not quite a nice solution. I don't like doing
>>>> patchwork like this, but i had to.
>>>>
>>>> The behavior i noticed was when the php started dieing, then all php
>>>> processes would be in "pfault" or "sbwait" state.
>>>>
>>>> If you need any more infos i will be happy to provide. i just want to
>>>> know what may be the cause of this.
>>>>
>>>> Information:
>>>> 1 - This is what nginx_status page shows when php is close to dieing.
>>>>
>>>> Active connections: 5018
>>>> server accepts handled requests
>>>> 11836 11836 28147
>>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>>
>>>> 2 - my php-fpm.conf
>>>>
>>>>
>>>> <?xml version="1.0" ?>
>>>> <configuration>
>>>>
>>>> All relative paths in this config are relative to php's
>>>> install prefix
>>>>
>>>> <section name="global_options">
>>>>
>>>> Pid file
>>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>>> fpm.pid</value>
>>>>
>>>> Error log file
>>>> <value name="error_log">/usr/local/kphp/logs/php-
>>>> fpm.log</value>
>>>>
>>>> Log level
>>>> <value name="log_level">notice</value>
>>>>
>>>> When this amount of php processes exited with SIGSEGV
>>>> or SIGBUS ...
>>>> <value name="emergency_restart_threshold">10</value>
>>>>
>>>> ... in a less than this interval of time, a graceful
>>>> restart will be initiated.
>>>> Useful to work around accidental curruptions in
>>>> accelerator's shared memory.
>>>> <value name="emergency_restart_interval">1m</value>
>>>>
>>>> Time limit on waiting child's reaction on signals from
>>>> master
>>>> <value name="process_control_timeout">5s</value>
>>>>
>>>> Set to 'no' to debug fpm
>>>> <value name="daemonize">yes</value>
>>>>
>>>> </section>
>>>>
>>>> <workers>
>>>>
>>>> <section name="pool">
>>>>
>>>> Name of pool. Used in logs and stats.
>>>> <value name="name">nginx</value>
>>>>
>>>> Address to accept fastcgi requests on.
>>>> Valid syntax is 'ip.ad.re.ss:port' or just
>>>> 'port' or '/path/to/unix/socket'
>>>> <value name="listen_address">127.0.0.1:8000</
>>>> value>
>>>>
>>>> <value name="listen_options">
>>>>
>>>> Set listen(2) backlog
>>>> <value name="backlog">-1</value>
>>>>
>>>> Set permissions for unix socket, if
>>>> one used.
>>>> In Linux read/write permissions must
>>>> be set in order to allow connections from web server.
>>>> Many BSD-derrived systems allow
>>>> connections regardless of permissions.
>>>> <value name="owner"></value>
>>>> <value name="group"></value>
>>>> <value name="mode">0666</value>
>>>> </value>
>>>>
>>>> Additional php.ini defines, specific to this
>>>> pool of workers.
>>>> <value name="php_defines">
>>>> 
>>>> 
>>>> </value>
>>>>
>>>> Unix user of processes
>>>> <value name="user">nobody</value>
>>>>
>>>> Unix group of processes
>>>> <value name="group">nobody</value>
>>>>
>>>> Process manager settings
>>>> <value name="pm">
>>>>
>>>> Sets style of controling worker
>>>> process count.
>>>> Valid values are 'static' and 'apache-
>>>> like'
>>>> <value name="style">static</value>
>>>>
>>>> Sets the limit on the number of
>>>> simultaneous requests that will be served.
>>>> Equivalent to Apache MaxClients
>>>> directive.
>>>> Equivalent to PHP_FCGI_CHILDREN
>>>> environment in original php.fcgi
>>>> Used with any pm_style.
>>>> <value name="max_children">256</value>
>>>>
>>>> Settings group for 'apache-like' pm
>>>> style
>>>> <value name="apache_like">
>>>>
>>>> Sets the number of server
>>>> processes created on startup.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value name="StartServers">32</
>>>> value>
>>>>
>>>> Sets the desired minimum
>>>> number of idle server processes.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MinSpareServers">64</value>
>>>>
>>>> Sets the desired maximum
>>>> number of idle server processes.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MaxSpareServers">128</value>
>>>>
>>>> </value>
>>>>
>>>> </value>
>>>>
>>>> The timeout (in seconds) for serving a single
>>>> request after which the worker process will be terminated
>>>> Should be used when 'max_execution_time' ini
>>>> option does not stop script execution for some reason
>>>> '0s' means 'off'
>>>> <value name="request_terminate_timeout">0s</
>>>> value>
>>>>
>>>> The timeout (in seconds) for serving of single
>>>> request after which a php backtrace will be dumped to slow.log file
>>>> '0s' means 'off'
>>>> <value name="request_slowlog_timeout">10s</
>>>> value>
>>>>
>>>> The log file for slow requests
>>>> <value name="slowlog">logs/slow.log</value>
>>>>
>>>> Set open file desc rlimit
>>>> <value name="rlimit_files">8092</value>
>>>>
>>>> Set max core size rlimit
>>>> <value name="rlimit_core">0</value>
>>>>
>>>> Chroot to this directory at the start,
>>>> absolute path
>>>> <value name="chroot"></value>
>>>>
>>>> Chdir to this directory at the start, absolute
>>>> path
>>>> <value name="chdir"></value>
>>>>
>>>> Redirect workers' stdout and stderr into main
>>>> error log.
>>>> If not set, they will be redirected to /dev/
>>>> null, according to FastCGI specs
>>>> <value name="catch_workers_output">yes</value>
>>>>
>>>> How much requests each process should execute
>>>> before respawn.
>>>> Useful to work around memory leaks in 3rd
>>>> party libraries.
>>>> For endless request processing please specify
>>>> 0
>>>> Equivalent to PHP_FCGI_MAX_REQUESTS
>>>> <value name="max_requests">500</value>
>>>>
>>>> Comma separated list of ipv4 addresses of
>>>> FastCGI clients that allowed to connect.
>>>> Equivalent to FCGI_WEB_SERVER_ADDRS
>>>> environment in original php.fcgi (5.2.2+)
>>>> Makes sense only with AF_INET listening
>>>> socket.
>>>> <value name="allowed_clients">127.0.0.1</
>>>> value>
>>>>
>>>> Pass environment variables like
>>>> LD_LIBRARY_PATH
>>>> All $VARIABLEs are taken from current
>>>> environment
>>>> <value name="environment">
>>>> <value name="HOSTNAME">$HOSTNAME</
>>>> value>
>>>> <value name="PATH">/usr/local/bin:/usr/
>>>> bin:/bin</value>
>>>> <value name="TMP">/tmp</value>
>>>> <value name="TMPDIR">/tmp</value>
>>>> <value name="TEMP">/tmp</value>
>>>> <value name="OSTYPE">$OSTYPE</value>
>>>> <value name="MACHTYPE">$MACHTYPE</
>>>> value>
>>>> <value name="MALLOC_CHECK_">2</value>
>>>> </value>
>>>>
>>>> </section>
>>>>
>>>> </workers>
>>>>
>>>> </configuration>
>>>>
>>>> 3 - the relevant portion of nginx config:
>>>>
>>>> user nobody;
>>>> worker_processes 64;
>>>> worker_rlimit_nofile 10240;
>>>>
>>>> error_log logs/error.log;
>>>> pid logs/nginx.pid;
>>>>
>>>> events {
>>>> worker_connections 8192;
>>>> use kqueue;
>>>> }
>>>>
>>>> [...............................]
>>>>
>>>> location ~* \.php$ {
>>>> fastcgi_pass 127.0.0.1:8000;
>>>> fastcgi_index index.php;
>>>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>>>> public_html$fastcgi_script_name;
>>>> fastcgi_param SCRIPT_NAME
>>>> $fastcgi_script_name;
>>>> fastcgi_param REQUEST_URI $request_uri;
>>>> fastcgi_param DOCUMENT_URI $document_uri;
>>>> fastcgi_param DOCUMENT_ROOT $document_root;
>>>> fastcgi_param QUERY_STRING $query_string;
>>>> fastcgi_param REQUEST_METHOD $request_method;
>>>> fastcgi_param CONTENT_TYPE $content_type;
>>>> fastcgi_param CONTENT_LENGTH $content_length;
>>>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>>>> fastcgi_param SERVER_SOFTWARE nginx;
>>>> fastcgi_param REMOTE_ADDR $remote_addr;
>>>> fastcgi_param REMOTE_PORT $remote_port;
>>>> fastcgi_param SERVER_ADDR $server_addr;
>>>> fastcgi_param SERVER_PORT $server_port;
>>>> fastcgi_param SERVER_NAME $server_name;
>>>> fastcgi_param SERVER_PROTOCOL
>>>> $server_protocol;
>>>>
>>>> fastcgi_buffers 64 64k;
>>>> fastcgi_buffer_size 64k;
>>>>
>>>
>>>
>>>
>>
>>
>
>

Reply Quote

iberkner

Re: Nginx high load - php stops at some point
September 28, 2010 02:42PM

Registered: 15 years ago
Posts: 156

We run a very highload site using Nginx and PHP 5.3.3 (with FPM) with
eAccelerator.

We are consistently at 5000+ connections, during peak, we've hit 7,000+.

I'm always finding stability issues due to the high-load but usually can
locate areas in the code base which cause PHP to hang or wait. In
particular, check database queries. We've had to separate our data requests
into a master->read only slave configuration.

Do you have your slow query log turned on and have you looked at what's in
there?

We're also going to be caching a lot of database calls. The idea is to
reduce the load and wait time on the PHP app servers.

On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
themediaplanet90210@gmail.com> wrote:

> I tried APC prior to eAccelerator, and i had the same issues. It actually
> happened more often for php to hang than now with eaccelerator.
>
> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>
> I was experiencing similar issues recently.
> Changed my opcode cache to APC, haven't seen the same problem since.
>
> Try APC and see if you still have the same issues.
>
> -Jon
>
> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>
>> Hi guys,
>>
>> I am running a very busy website. It's running a photo gallery ->
>> Gallery2.
>>
>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>
>> At some point during the day, pretty much random, the load on the
>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>> cache.
>>
>> I have been debugging and i noticed that when the server reaches over
>> 6000 active connections, then everything gets clogged and eventually i
>> had to build a script that would restart php/nginx when the load
>> starts spiking, which is not quite a nice solution. I don't like doing
>> patchwork like this, but i had to.
>>
>> The behavior i noticed was when the php started dieing, then all php
>> processes would be in "pfault" or "sbwait" state.
>>
>> If you need any more infos i will be happy to provide. i just want to
>> know what may be the cause of this.
>>
>> Information:
>> 1 - This is what nginx_status page shows when php is close to dieing.
>>
>> Active connections: 5018
>> server accepts handled requests
>> 11836 11836 28147
>> Reading: 49 Writing: 2103 Waiting: 2866
>>
>> 2 - my php-fpm.conf
>>
>>
>> <?xml version="1.0" ?>
>> <configuration>
>>
>> All relative paths in this config are relative to php's
>> install prefix
>>
>> <section name="global_options">
>>
>> Pid file
>> <value name="pid_file">/usr/local/kphp/logs/php-
>> fpm.pid</value>
>>
>> Error log file
>> <value name="error_log">/usr/local/kphp/logs/php-
>> fpm.log</value>
>>
>> Log level
>> <value name="log_level">notice</value>
>>
>> When this amount of php processes exited with SIGSEGV
>> or SIGBUS ...
>> <value name="emergency_restart_threshold">10</value>
>>
>> ... in a less than this interval of time, a graceful
>> restart will be initiated.
>> Useful to work around accidental curruptions in
>> accelerator's shared memory.
>> <value name="emergency_restart_interval">1m</value>
>>
>> Time limit on waiting child's reaction on signals from
>> master
>> <value name="process_control_timeout">5s</value>
>>
>> Set to 'no' to debug fpm
>> <value name="daemonize">yes</value>
>>
>> </section>
>>
>> <workers>
>>
>> <section name="pool">
>>
>> Name of pool. Used in logs and stats.
>> <value name="name">nginx</value>
>>
>> Address to accept fastcgi requests on.
>> Valid syntax is 'ip.ad.re.ss:port' or just
>> 'port' or '/path/to/unix/socket'
>> <value name="listen_address">127.0.0.1:8000</
>> value>
>>
>> <value name="listen_options">
>>
>> Set listen(2) backlog
>> <value name="backlog">-1</value>
>>
>> Set permissions for unix socket, if
>> one used.
>> In Linux read/write permissions must
>> be set in order to allow connections from web server.
>> Many BSD-derrived systems allow
>> connections regardless of permissions.
>> <value name="owner"></value>
>> <value name="group"></value>
>> <value name="mode">0666</value>
>> </value>
>>
>> Additional php.ini defines, specific to this
>> pool of workers.
>> <value name="php_defines">
>> 
>> 
>> </value>
>>
>> Unix user of processes
>> <value name="user">nobody</value>
>>
>> Unix group of processes
>> <value name="group">nobody</value>
>>
>> Process manager settings
>> <value name="pm">
>>
>> Sets style of controling worker
>> process count.
>> Valid values are 'static' and 'apache-
>> like'
>> <value name="style">static</value>
>>
>> Sets the limit on the number of
>> simultaneous requests that will be served.
>> Equivalent to Apache MaxClients
>> directive.
>> Equivalent to PHP_FCGI_CHILDREN
>> environment in original php.fcgi
>> Used with any pm_style.
>> <value name="max_children">256</value>
>>
>> Settings group for 'apache-like' pm
>> style
>> <value name="apache_like">
>>
>> Sets the number of server
>> processes created on startup.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value name="StartServers">32</
>> value>
>>
>> Sets the desired minimum
>> number of idle server processes.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value
>> name="MinSpareServers">64</value>
>>
>> Sets the desired maximum
>> number of idle server processes.
>> Used only when 'apache-like'
>> pm_style is selected
>> <value
>> name="MaxSpareServers">128</value>
>>
>> </value>
>>
>> </value>
>>
>> The timeout (in seconds) for serving a single
>> request after which the worker process will be terminated
>> Should be used when 'max_execution_time' ini
>> option does not stop script execution for some reason
>> '0s' means 'off'
>> <value name="request_terminate_timeout">0s</
>> value>
>>
>> The timeout (in seconds) for serving of single
>> request after which a php backtrace will be dumped to slow.log file
>> '0s' means 'off'
>> <value name="request_slowlog_timeout">10s</
>> value>
>>
>> The log file for slow requests
>> <value name="slowlog">logs/slow.log</value>
>>
>> Set open file desc rlimit
>> <value name="rlimit_files">8092</value>
>>
>> Set max core size rlimit
>> <value name="rlimit_core">0</value>
>>
>> Chroot to this directory at the start,
>> absolute path
>> <value name="chroot"></value>
>>
>> Chdir to this directory at the start, absolute
>> path
>> <value name="chdir"></value>
>>
>> Redirect workers' stdout and stderr into main
>> error log.
>> If not set, they will be redirected to /dev/
>> null, according to FastCGI specs
>> <value name="catch_workers_output">yes</value>
>>
>> How much requests each process should execute
>> before respawn.
>> Useful to work around memory leaks in 3rd
>> party libraries.
>> For endless request processing please specify
>> 0
>> Equivalent to PHP_FCGI_MAX_REQUESTS
>> <value name="max_requests">500</value>
>>
>> Comma separated list of ipv4 addresses of
>> FastCGI clients that allowed to connect.
>> Equivalent to FCGI_WEB_SERVER_ADDRS
>> environment in original php.fcgi (5.2.2+)
>> Makes sense only with AF_INET listening
>> socket.
>> <value name="allowed_clients">127.0.0.1</
>> value>
>>
>> Pass environment variables like
>> LD_LIBRARY_PATH
>> All $VARIABLEs are taken from current
>> environment
>> <value name="environment">
>> <value name="HOSTNAME">$HOSTNAME</
>> value>
>> <value name="PATH">/usr/local/bin:/usr/
>> bin:/bin</value>
>> <value name="TMP">/tmp</value>
>> <value name="TMPDIR">/tmp</value>
>> <value name="TEMP">/tmp</value>
>> <value name="OSTYPE">$OSTYPE</value>
>> <value name="MACHTYPE">$MACHTYPE</
>> value>
>> <value name="MALLOC_CHECK_">2</value>
>> </value>
>>
>> </section>
>>
>> </workers>
>>
>> </configuration>
>>
>> 3 - the relevant portion of nginx config:
>>
>> user nobody;
>> worker_processes 64;
>> worker_rlimit_nofile 10240;
>>
>> error_log logs/error.log;
>> pid logs/nginx.pid;
>>
>> events {
>> worker_connections 8192;
>> use kqueue;
>> }
>>
>> [...............................]
>>
>> location ~* \.php$ {
>> fastcgi_pass 127.0.0.1:8000;
>> fastcgi_index index.php;
>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>> public_html$fastcgi_script_name;
>> fastcgi_param SCRIPT_NAME
>> $fastcgi_script_name;
>> fastcgi_param REQUEST_URI $request_uri;
>> fastcgi_param DOCUMENT_URI $document_uri;
>> fastcgi_param DOCUMENT_ROOT $document_root;
>> fastcgi_param QUERY_STRING $query_string;
>> fastcgi_param REQUEST_METHOD $request_method;
>> fastcgi_param CONTENT_TYPE $content_type;
>> fastcgi_param CONTENT_LENGTH $content_length;
>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>> fastcgi_param SERVER_SOFTWARE nginx;
>> fastcgi_param REMOTE_ADDR $remote_addr;
>> fastcgi_param REMOTE_PORT $remote_port;
>> fastcgi_param SERVER_ADDR $server_addr;
>> fastcgi_param SERVER_PORT $server_port;
>> fastcgi_param SERVER_NAME $server_name;
>> fastcgi_param SERVER_PROTOCOL
>> $server_protocol;
>>
>> fastcgi_buffers 64 64k;
>> fastcgi_buffer_size 64k;
>>
>
>
>

Reply Quote

c2h5oh

Re: Nginx high load - php stops at some point
September 28, 2010 02:52PM

I've had a similar looking problem a while ego and to my surprise the
culprit was MySQL connection limit.
Once the connection limit was reached php processes became
unresponsive one after another.

I didn't have enough time to get into details why connection didn't
just fail and return an error but increasing connection limit solved
it.

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 28, 2010 02:56PM

Ok i ran siege, with 200 concurrent connections / internet = true, and
i gave it a list of urls to access. After about 3 4 minutes, php
processes are in sbwait state, and the final message is gateway timeout.
The only thing i could get from php-fpm.conf. No other error messages.

Sep 29 02:22:13.203711 [NOTICE] fpm_got_signal(), line 48: received SIGCHLD
Sep 29 02:22:13.383918 [WARNING] fpm_children_bury(), line 215: child
82480 (pool nginx) exited on signal 9 SIGKILL after 664.244869 seconds
from start

On 9/28/2010 8:25 PM, Jonathan Langevin wrote:
> You may be able to replicate the problem by running siege against the
> webserver.
> I typically set up a known heavy PHP page (you can also provide a list
> of urls, accessed randomly) to be the target. It's good at least for
> getting an idea of how many concurrent connections you can handle.
>
> http://www.joedog.org/index/siege-home
>
> On Tue, Sep 28, 2010 at 2:16 PM, StefanitaRares Dumitrescu
> <themediaplanet90210@gmail.com <mailto:themediaplanet90210@gmail.com>>
> wrote:
>
> I upgraded to the latest php 5.2 - 5.2.14 with php-fpm 0.5.14,
> just to take the "old version thing" out of equation.
>
> I have made a few small modifications, to the php-fpm.conf, i
> increased the number of minservers and max servers and start
> servers. The traffic is lower now so i cannot really know for sure
> this is works, as the php crash/hang usually happens during the day.
>
> I also decreased the number of nginx workers to 8 instead of 64,
> as i think it made no sense to have 64, considering i have 4 cores
> with HT.
>
>
> Used with any pm_style.
> <value name="max_children">256</value>
> Settings group for 'apache-like'
> pm style
> <value name="apache_like">
> Sets the number of server
> processes created on startup.
> Used only when
> 'apache-like' pm_style is selected
> <value name="StartServers">64</value>
>
> Sets the desired minimum
> number of idle server processes.
> Used only when
> 'apache-like' pm_style is selected
> <value name="MinSpareServers">128</value>
>
> Sets the desired maximum
> number of idle server processes.
> Used only when
> 'apache-like' pm_style is selected
> <value name="MaxSpareServers">256</value>
> </value>
> </value>
>
>
>
> On 9/28/2010 7:38 PM, Jonathan Langevin wrote:
>> My apologies on versions, I missed that in my follow-up.
>>
>> On Tue, Sep 28, 2010 at 1:37 PM, StefanitaRares Dumitrescu
>> <themediaplanet90210@gmail.com
>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>
>> I already specified the versions of everything in the first
>> post. php 5.2.10 / nginx latest - 0.8.52.
>>
>> I am going to check the debug options about now.
>>
>> On 9/28/2010 7:31 PM, Jonathan Langevin wrote:
>>> What versions of nginx, php, and fpm are you running?
>>> It appears to be older versions, due to the configurations
>>> that you have in place.
>>>
>>> Try setting (in FPM config):
>>> catch_workers_output = yes
>>> log_level = debug
>>>
>>> I believe your version of FPM will have to have been
>>> compiled with debugging enabled for that log level to function.
>>> I'm also not certain catch_workers_output will function for
>>> you or not, but I do know that since that setting has been
>>> enabled (which I changed at the same time that I changed to
>>> APC), I have not seen the same errors again.
>>>
>>> Be warned, debug log level is extremely verbose, expect logs
>>> to grow exponentially.
>>>
>>> -Jon
>>>
>>> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu
>>> <themediaplanet90210@gmail.com
>>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>>
>>> I tried APC prior to eAccelerator, and i had the same
>>> issues. It actually happened more often for php to hang
>>> than now with eaccelerator.
>>>
>>>
>>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>>> I was experiencing similar issues recently.
>>>> Changed my opcode cache to APC, haven't seen the same
>>>> problem since.
>>>>
>>>> Try APC and see if you still have the same issues.
>>>>
>>>> -Jon
>>>>
>>>> On Tue, Sep 28, 2010 at 1:15 PM, katmai
>>>> <themediaplanet90210@gmail.com
>>>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> I am running a very busy website. It's running a
>>>> photo gallery ->
>>>> Gallery2.
>>>>
>>>> The system is a Quad core Xeon, with 16 gb RAM, and
>>>> separate drives
>>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch
>>>> level.
>>>>
>>>> At some point during the day, pretty much random,
>>>> the load on the
>>>> server spikes, and then php processing stops. I run
>>>> nginx 0.8.52, and
>>>> php 5.2.10 with fpm patch, and i am using
>>>> eAccelerator as an opcode
>>>> cache.
>>>>
>>>> I have been debugging and i noticed that when the
>>>> server reaches over
>>>> 6000 active connections, then everything gets
>>>> clogged and eventually i
>>>> had to build a script that would restart php/nginx
>>>> when the load
>>>> starts spiking, which is not quite a nice solution.
>>>> I don't like doing
>>>> patchwork like this, but i had to.
>>>>
>>>> The behavior i noticed was when the php started
>>>> dieing, then all php
>>>> processes would be in "pfault" or "sbwait" state.
>>>>
>>>> If you need any more infos i will be happy to
>>>> provide. i just want to
>>>> know what may be the cause of this.
>>>>
>>>> Information:
>>>> 1 - This is what nginx_status page shows when php
>>>> is close to dieing.
>>>>
>>>> Active connections: 5018
>>>> server accepts handled requests
>>>> 11836 11836 28147
>>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>>
>>>> 2 - my php-fpm.conf
>>>>
>>>>
>>>> <?xml version="1.0" ?>
>>>> <configuration>
>>>>
>>>> All relative paths in this config are
>>>> relative to php's
>>>> install prefix
>>>>
>>>> <section name="global_options">
>>>>
>>>> Pid file
>>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>>> fpm.pid</value>
>>>>
>>>> Error log file
>>>> <value name="error_log">/usr/local/kphp/logs/php-
>>>> fpm.log</value>
>>>>
>>>> Log level
>>>> <value name="log_level">notice</value>
>>>>
>>>> When this amount of php processes
>>>> exited with SIGSEGV
>>>> or SIGBUS ...
>>>> <value name="emergency_restart_threshold">10</value>
>>>>
>>>> ... in a less than this interval of
>>>> time, a graceful
>>>> restart will be initiated.
>>>> Useful to work around accidental
>>>> curruptions in
>>>> accelerator's shared memory.
>>>> <value name="emergency_restart_interval">1m</value>
>>>>
>>>> Time limit on waiting child's
>>>> reaction on signals from
>>>> master
>>>> <value name="process_control_timeout">5s</value>
>>>>
>>>> Set to 'no' to debug fpm
>>>> <value name="daemonize">yes</value>
>>>>
>>>> </section>
>>>>
>>>> <workers>
>>>>
>>>> <section name="pool">
>>>>
>>>> Name of pool. Used in logs
>>>> and stats.
>>>> <value name="name">nginx</value>
>>>>
>>>> Address to accept fastcgi
>>>> requests on.
>>>> Valid syntax is
>>>> 'ip.ad.re.ss:port' or just
>>>> 'port' or '/path/to/unix/socket'
>>>> <value name="listen_address">127.0.0.1:8000
>>>> http://127.0.0.1:8000</
>>>> value>
>>>>
>>>> <value name="listen_options">
>>>>
>>>> Set listen(2) backlog
>>>> <value name="backlog">-1</value>
>>>>
>>>> Set permissions for
>>>> unix socket, if
>>>> one used.
>>>> In Linux read/write
>>>> permissions must
>>>> be set in order to allow connections from web server.
>>>> Many BSD-derrived
>>>> systems allow
>>>> connections regardless of permissions.
>>>> <value name="owner"></value>
>>>> <value name="group"></value>
>>>> <value name="mode">0666</value>
>>>> </value>
>>>>
>>>> Additional php.ini defines,
>>>> specific to this
>>>> pool of workers.
>>>> <value name="php_defines">
>>>> 
>>>> 
>>>> </value>
>>>>
>>>> Unix user of processes
>>>> <value name="user">nobody</value>
>>>>
>>>> Unix group of processes
>>>> <value name="group">nobody</value>
>>>>
>>>> Process manager settings
>>>> <value name="pm">
>>>>
>>>> Sets style of
>>>> controling worker
>>>> process count.
>>>> Valid values are
>>>> 'static' and 'apache-
>>>> like'
>>>> <value name="style">static</value>
>>>>
>>>> Sets the limit on
>>>> the number of
>>>> simultaneous requests that will be served.
>>>> Equivalent to Apache
>>>> MaxClients
>>>> directive.
>>>> Equivalent to
>>>> PHP_FCGI_CHILDREN
>>>> environment in original php.fcgi
>>>> Used with any pm_style.
>>>> <value name="max_children">256</value>
>>>>
>>>> Settings group for
>>>> 'apache-like' pm
>>>> style
>>>> <value name="apache_like">
>>>>
>>>> Sets the
>>>> number of server
>>>> processes created on startup.
>>>> Used only
>>>> when 'apache-like'
>>>> pm_style is selected
>>>> <value name="StartServers">32</
>>>> value>
>>>>
>>>> Sets the
>>>> desired minimum
>>>> number of idle server processes.
>>>> Used only
>>>> when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MinSpareServers">64</value>
>>>>
>>>> Sets the
>>>> desired maximum
>>>> number of idle server processes.
>>>> Used only
>>>> when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MaxSpareServers">128</value>
>>>>
>>>> </value>
>>>>
>>>> </value>
>>>>
>>>> The timeout (in seconds) for
>>>> serving a single
>>>> request after which the worker process will be
>>>> terminated
>>>> Should be used when
>>>> 'max_execution_time' ini
>>>> option does not stop script execution for some reason
>>>> '0s' means 'off'
>>>> <value name="request_terminate_timeout">0s</
>>>> value>
>>>>
>>>> The timeout (in seconds) for
>>>> serving of single
>>>> request after which a php backtrace will be dumped
>>>> to slow.log file
>>>> '0s' means 'off'
>>>> <value name="request_slowlog_timeout">10s</
>>>> value>
>>>>
>>>> The log file for slow requests
>>>> <value name="slowlog">logs/slow.log</value>
>>>>
>>>> Set open file desc rlimit
>>>> <value name="rlimit_files">8092</value>
>>>>
>>>> Set max core size rlimit
>>>> <value name="rlimit_core">0</value>
>>>>
>>>> Chroot to this directory at
>>>> the start,
>>>> absolute path
>>>> <value name="chroot"></value>
>>>>
>>>> Chdir to this directory at
>>>> the start, absolute
>>>> path
>>>> <value name="chdir"></value>
>>>>
>>>> Redirect workers' stdout and
>>>> stderr into main
>>>> error log.
>>>> If not set, they will be
>>>> redirected to /dev/
>>>> null, according to FastCGI specs
>>>> <value name="catch_workers_output">yes</value>
>>>>
>>>> How much requests each
>>>> process should execute
>>>> before respawn.
>>>> Useful to work around memory
>>>> leaks in 3rd
>>>> party libraries.
>>>> For endless request
>>>> processing please specify
>>>> 0
>>>> Equivalent to
>>>> PHP_FCGI_MAX_REQUESTS
>>>> <value name="max_requests">500</value>
>>>>
>>>> Comma separated list of ipv4
>>>> addresses of
>>>> FastCGI clients that allowed to connect.
>>>> Equivalent to
>>>> FCGI_WEB_SERVER_ADDRS
>>>> environment in original php.fcgi (5.2.2+)
>>>> Makes sense only with
>>>> AF_INET listening
>>>> socket.
>>>> <value name="allowed_clients">127.0.0.1</
>>>> value>
>>>>
>>>> Pass environment variables like
>>>> LD_LIBRARY_PATH
>>>> All $VARIABLEs are taken
>>>> from current
>>>> environment
>>>> <value name="environment">
>>>> <value name="HOSTNAME">$HOSTNAME</
>>>> value>
>>>> <value name="PATH">/usr/local/bin:/usr/
>>>> bin:/bin</value>
>>>> <value name="TMP">/tmp</value>
>>>> <value name="TMPDIR">/tmp</value>
>>>> <value name="TEMP">/tmp</value>
>>>> <value name="OSTYPE">$OSTYPE</value>
>>>> <value name="MACHTYPE">$MACHTYPE</
>>>> value>
>>>> <value name="MALLOC_CHECK_">2</value>
>>>> </value>
>>>>
>>>> </section>
>>>>
>>>> </workers>
>>>>
>>>> </configuration>
>>>>
>>>> 3 - the relevant portion of nginx config:
>>>>
>>>> user nobody;
>>>> worker_processes 64;
>>>> worker_rlimit_nofile 10240;
>>>>
>>>> error_log logs/error.log;
>>>> pid logs/nginx.pid;
>>>>
>>>> events {
>>>> worker_connections 8192;
>>>> use kqueue;
>>>> }
>>>>
>>>> [...............................]
>>>>
>>>> location ~* \.php$ {
>>>> fastcgi_pass 127.0.0.1:8000
>>>> http://127.0.0.1:8000;
>>>> fastcgi_index index.php;
>>>> fastcgi_param SCRIPT_FILENAME
>>>> /xdrive/cityv3/
>>>> public_html$fastcgi_script_name;
>>>> fastcgi_param SCRIPT_NAME
>>>> $fastcgi_script_name;
>>>> fastcgi_param REQUEST_URI
>>>> $request_uri;
>>>> fastcgi_param DOCUMENT_URI
>>>> $document_uri;
>>>> fastcgi_param DOCUMENT_ROOT
>>>> $document_root;
>>>> fastcgi_param QUERY_STRING
>>>> $query_string;
>>>> fastcgi_param REQUEST_METHOD
>>>> $request_method;
>>>> fastcgi_param CONTENT_TYPE
>>>> $content_type;
>>>> fastcgi_param CONTENT_LENGTH
>>>> $content_length;
>>>> fastcgi_param GATEWAY_INTERFACE
>>>> CGI/1.1;
>>>> fastcgi_param SERVER_SOFTWARE
>>>> nginx;
>>>> fastcgi_param REMOTE_ADDR
>>>> $remote_addr;
>>>> fastcgi_param REMOTE_PORT
>>>> $remote_port;
>>>> fastcgi_param SERVER_ADDR
>>>> $server_addr;
>>>> fastcgi_param SERVER_PORT
>>>> $server_port;
>>>> fastcgi_param SERVER_NAME
>>>> $server_name;
>>>> fastcgi_param SERVER_PROTOCOL
>>>> $server_protocol;
>>>>
>>>> fastcgi_buffers 64 64k;
>>>> fastcgi_buffer_size 64k;
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 28, 2010 02:58PM

Actually i think i modified my config at some point and took out slow
log from sql. I need to fix this now. Will see where i go with it.

Regarding the connection limit for SQL - i have it at 2000. Should be
more than enough imho.

On 9/28/2010 7:28 PM, Ilan Berkner wrote:
> We run a very highload site using Nginx and PHP 5.3.3 (with FPM) with
> eAccelerator.
>
> We are consistently at 5000+ connections, during peak, we've hit 7,000+.
>
> I'm always finding stability issues due to the high-load but usually
> can locate areas in the code base which cause PHP to hang or wait. In
> particular, check database queries. We've had to separate our data
> requests into a master->read only slave configuration.
>
> Do you have your slow query log turned on and have you looked at
> what's in there?
>
> We're also going to be caching a lot of database calls. The idea is
> to reduce the load and wait time on the PHP app servers.
>
> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu
> <themediaplanet90210@gmail.com <mailto:themediaplanet90210@gmail.com>>
> wrote:
>
> I tried APC prior to eAccelerator, and i had the same issues. It
> actually happened more often for php to hang than now with
> eaccelerator.
>
> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>> I was experiencing similar issues recently.
>> Changed my opcode cache to APC, haven't seen the same problem since.
>>
>> Try APC and see if you still have the same issues.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:15 PM, katmai
>> <themediaplanet90210@gmail.com
>> <mailto:themediaplanet90210@gmail.com>> wrote:
>>
>> Hi guys,
>>
>> I am running a very busy website. It's running a photo gallery ->
>> Gallery2.
>>
>> The system is a Quad core Xeon, with 16 gb RAM, and separate
>> drives
>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>
>> At some point during the day, pretty much random, the load on the
>> server spikes, and then php processing stops. I run nginx
>> 0.8.52, and
>> php 5.2.10 with fpm patch, and i am using eAccelerator as an
>> opcode
>> cache.
>>
>> I have been debugging and i noticed that when the server
>> reaches over
>> 6000 active connections, then everything gets clogged and
>> eventually i
>> had to build a script that would restart php/nginx when the load
>> starts spiking, which is not quite a nice solution. I don't
>> like doing
>> patchwork like this, but i had to.
>>
>> The behavior i noticed was when the php started dieing, then
>> all php
>> processes would be in "pfault" or "sbwait" state.
>>
>> If you need any more infos i will be happy to provide. i just
>> want to
>> know what may be the cause of this.
>>
>> Information:
>> 1 - This is what nginx_status page shows when php is close to
>> dieing.
>>
>> Active connections: 5018
>> server accepts handled requests
>> 11836 11836 28147
>> Reading: 49 Writing: 2103 Waiting: 2866
>>
>> 2 - my php-fpm.conf
>>
>>
>> <?xml version="1.0" ?>
>> <configuration>
>>
>> All relative paths in this config are relative to php's
>> install prefix
>>
>> <section name="global_options">
>>
>> Pid file
>> <value name="pid_file">/usr/local/kphp/logs/php-
>> fpm.pid</value>
>>
>> Error log file
>> <value name="error_log">/usr/local/kphp/logs/php-
>> fpm.log</value>
>>
>> Log level
>> <value name="log_level">notice</value>
>>
>> When this amount of php processes exited with
>> SIGSEGV
>> or SIGBUS ...
>> <value name="emergency_restart_threshold">10</value>
>>
>> ... in a less than this interval of time, a
>> graceful
>> restart will be initiated.
>> Useful to work around accidental curruptions in
>> accelerator's shared memory.
>> <value name="emergency_restart_interval">1m</value>
>>
>> Time limit on waiting child's reaction on
>> signals from
>> master
>> <value name="process_control_timeout">5s</value>
>>
>> Set to 'no' to debug fpm
>> <value name="daemonize">yes</value>
>>
>> </section>
>>
>> <workers>
>>
>> <section name="pool">
>>
>> Name of pool. Used in logs and stats.
>> <value name="name">nginx</value>
>>
>> Address to accept fastcgi requests on.
>> Valid syntax is 'ip.ad.re.ss:port' or just
>> 'port' or '/path/to/unix/socket'
>> <value name="listen_address">127.0.0.1:8000
>> http://127.0.0.1:8000</
>> value>
>>
>> <value name="listen_options">
>>
>> Set listen(2) backlog
>> <value name="backlog">-1</value>
>>
>> Set permissions for unix
>> socket, if
>> one used.
>> In Linux read/write
>> permissions must
>> be set in order to allow connections from web server.
>> Many BSD-derrived systems allow
>> connections regardless of permissions.
>> <value name="owner"></value>
>> <value name="group"></value>
>> <value name="mode">0666</value>
>> </value>
>>
>> Additional php.ini defines, specific
>> to this
>> pool of workers.
>> <value name="php_defines">
>> 
>> 
>> </value>
>>
>> Unix user of processes
>> <value name="user">nobody</value>
>>
>> Unix group of processes
>> <value name="group">nobody</value>
>>
>> Process manager settings
>> <value name="pm">
>>
>> Sets style of controling worker
>> process count.
>> Valid values are 'static' and
>> 'apache-
>> like'
>> <value name="style">static</value>
>>
>> Sets the limit on the number of
>> simultaneous requests that will be served.
>> Equivalent to Apache MaxClients
>> directive.
>> Equivalent to PHP_FCGI_CHILDREN
>> environment in original php.fcgi
>> Used with any pm_style.
>> <value name="max_children">256</value>
>>
>> Settings group for
>> 'apache-like' pm
>> style
>> <value name="apache_like">
>>
>> Sets the number of server
>> processes created on startup.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value name="StartServers">32</
>> value>
>>
>> Sets the desired minimum
>> number of idle server processes.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value
>> name="MinSpareServers">64</value>
>>
>> Sets the desired maximum
>> number of idle server processes.
>> Used only when
>> 'apache-like'
>> pm_style is selected
>> <value
>> name="MaxSpareServers">128</value>
>>
>> </value>
>>
>> </value>
>>
>> The timeout (in seconds) for serving a
>> single
>> request after which the worker process will be terminated
>> Should be used when
>> 'max_execution_time' ini
>> option does not stop script execution for some reason
>> '0s' means 'off'
>> <value name="request_terminate_timeout">0s</
>> value>
>>
>> The timeout (in seconds) for serving
>> of single
>> request after which a php backtrace will be dumped to
>> slow.log file
>> '0s' means 'off'
>> <value name="request_slowlog_timeout">10s</
>> value>
>>
>> The log file for slow requests
>> <value name="slowlog">logs/slow.log</value>
>>
>> Set open file desc rlimit
>> <value name="rlimit_files">8092</value>
>>
>> Set max core size rlimit
>> <value name="rlimit_core">0</value>
>>
>> Chroot to this directory at the start,
>> absolute path
>> <value name="chroot"></value>
>>
>> Chdir to this directory at the start,
>> absolute
>> path
>> <value name="chdir"></value>
>>
>> Redirect workers' stdout and stderr
>> into main
>> error log.
>> If not set, they will be redirected to
>> /dev/
>> null, according to FastCGI specs
>> <value name="catch_workers_output">yes</value>
>>
>> How much requests each process should
>> execute
>> before respawn.
>> Useful to work around memory leaks in 3rd
>> party libraries.
>> For endless request processing please
>> specify
>> 0
>> Equivalent to PHP_FCGI_MAX_REQUESTS
>> <value name="max_requests">500</value>
>>
>> Comma separated list of ipv4 addresses of
>> FastCGI clients that allowed to connect.
>> Equivalent to FCGI_WEB_SERVER_ADDRS
>> environment in original php.fcgi (5.2.2+)
>> Makes sense only with AF_INET listening
>> socket.
>> <value name="allowed_clients">127.0.0.1</
>> value>
>>
>> Pass environment variables like
>> LD_LIBRARY_PATH
>> All $VARIABLEs are taken from current
>> environment
>> <value name="environment">
>> <value name="HOSTNAME">$HOSTNAME</
>> value>
>> <value name="PATH">/usr/local/bin:/usr/
>> bin:/bin</value>
>> <value name="TMP">/tmp</value>
>> <value name="TMPDIR">/tmp</value>
>> <value name="TEMP">/tmp</value>
>> <value name="OSTYPE">$OSTYPE</value>
>> <value name="MACHTYPE">$MACHTYPE</
>> value>
>> <value name="MALLOC_CHECK_">2</value>
>> </value>
>>
>> </section>
>>
>> </workers>
>>
>> </configuration>
>>
>> 3 - the relevant portion of nginx config:
>>
>> user nobody;
>> worker_processes 64;
>> worker_rlimit_nofile 10240;
>>
>> error_log logs/error.log;
>> pid logs/nginx.pid;
>>
>> events {
>> worker_connections 8192;
>> use kqueue;
>> }
>>
>> [...............................]
>>
>> location ~* \.php$ {
>> fastcgi_pass 127.0.0.1:8000
>> http://127.0.0.1:8000;
>> fastcgi_index index.php;
>> fastcgi_param SCRIPT_FILENAME
>> /xdrive/cityv3/
>> public_html$fastcgi_script_name;
>> fastcgi_param SCRIPT_NAME
>> $fastcgi_script_name;
>> fastcgi_param REQUEST_URI
>> $request_uri;
>> fastcgi_param DOCUMENT_URI
>> $document_uri;
>> fastcgi_param DOCUMENT_ROOT
>> $document_root;
>> fastcgi_param QUERY_STRING
>> $query_string;
>> fastcgi_param REQUEST_METHOD
>> $request_method;
>> fastcgi_param CONTENT_TYPE
>> $content_type;
>> fastcgi_param CONTENT_LENGTH
>> $content_length;
>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>> fastcgi_param SERVER_SOFTWARE nginx;
>> fastcgi_param REMOTE_ADDR
>> $remote_addr;
>> fastcgi_param REMOTE_PORT
>> $remote_port;
>> fastcgi_param SERVER_ADDR
>> $server_addr;
>> fastcgi_param SERVER_PORT
>> $server_port;
>> fastcgi_param SERVER_NAME
>> $server_name;
>> fastcgi_param SERVER_PROTOCOL
>> $server_protocol;
>>
>> fastcgi_buffers 64 64k;
>> fastcgi_buffer_size 64k;
>>
>>
>
>

Reply Quote

iberkner

Re: Nginx high load - php stops at some point
September 28, 2010 02:58PM

Registered: 15 years ago
Posts: 156

Another possible issue with php and mysql is dns resolution. for every
request, mysql has to do resolve the host name (forward and reverse), if
that fails or takes a long time, things can definitely get ugly and php
hangs waiting for responses, which eventually will return but really slowly
and cause a backup.

If DNS resolution is an issue, look into starting mysql with the
--skip-name-resolve option, all your hosts in the user table would have to
be ip addresses.

On Tue, Sep 28, 2010 at 2:50 PM, c2h5oh <maciej.lisiewski@gmail.com> wrote:

> I've had a similar looking problem a while ego and to my surprise the
> culprit was MySQL connection limit.
> Once the connection limit was reached php processes became
> unresponsive one after another.
>
> I didn't have enough time to get into details why connection didn't
> just fail and return an error but increasing connection limit solved
> it.

Reply Quote

Jonathan Langevin

Re: Nginx high load - php stops at some point
September 28, 2010 03:04PM

Considering the connection limit issue in MySQL, it's also possible your
MySQL connections are not terminating.
You can watch this through "mytop" (similar to top/htop, but for MySQL query
processes).

On siege, you can limit the amount of time to request (I forget the
parameter). This can help you with benchmarking, so that you can test within
a 1 minute benchmark, and keep increasing number of concurrent connections.
Then you also get a nice report summary from Siege, telling you the
availability % of your requests, request time, concurrency avg, etc.
Keep in mind there are limitations on concurrency within certain operating
systems, both outgoing and incoming.

On Tue, Sep 28, 2010 at 2:56 PM, StefanitaRares Dumitrescu <
themediaplanet90210@gmail.com> wrote:

> Actually i think i modified my config at some point and took out slow log
> from sql. I need to fix this now. Will see where i go with it.
>
> Regarding the connection limit for SQL - i have it at 2000. Should be more
> than enough imho.
>
>
> On 9/28/2010 7:28 PM, Ilan Berkner wrote:
>
> We run a very highload site using Nginx and PHP 5.3.3 (with FPM) with
> eAccelerator.
>
> We are consistently at 5000+ connections, during peak, we've hit 7,000+.
>
>
> I'm always finding stability issues due to the high-load but usually can
> locate areas in the code base which cause PHP to hang or wait. In
> particular, check database queries. We've had to separate our data requests
> into a master->read only slave configuration.
>
> Do you have your slow query log turned on and have you looked at what's
> in there?
>
> We're also going to be caching a lot of database calls. The idea is to
> reduce the load and wait time on the PHP app servers.
>
> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
> themediaplanet90210@gmail.com> wrote:
>
>> I tried APC prior to eAccelerator, and i had the same issues. It actually
>> happened more often for php to hang than now with eaccelerator.
>>
>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>
>> I was experiencing similar issues recently.
>> Changed my opcode cache to APC, haven't seen the same problem since.
>>
>> Try APC and see if you still have the same issues.
>>
>> -Jon
>>
>> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>>
>>> Hi guys,
>>>
>>> I am running a very busy website. It's running a photo gallery ->
>>> Gallery2.
>>>
>>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>>
>>> At some point during the day, pretty much random, the load on the
>>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>>> cache.
>>>
>>> I have been debugging and i noticed that when the server reaches over
>>> 6000 active connections, then everything gets clogged and eventually i
>>> had to build a script that would restart php/nginx when the load
>>> starts spiking, which is not quite a nice solution. I don't like doing
>>> patchwork like this, but i had to.
>>>
>>> The behavior i noticed was when the php started dieing, then all php
>>> processes would be in "pfault" or "sbwait" state.
>>>
>>> If you need any more infos i will be happy to provide. i just want to
>>> know what may be the cause of this.
>>>
>>> Information:
>>> 1 - This is what nginx_status page shows when php is close to dieing.
>>>
>>> Active connections: 5018
>>> server accepts handled requests
>>> 11836 11836 28147
>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>
>>> 2 - my php-fpm.conf
>>>
>>>
>>> <?xml version="1.0" ?>
>>> <configuration>
>>>
>>> All relative paths in this config are relative to php's
>>> install prefix
>>>
>>> <section name="global_options">
>>>
>>> Pid file
>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>> fpm.pid</value>
>>>
>>> Error log file
>>> <value name="error_log">/usr/local/kphp/logs/php-
>>> fpm.log</value>
>>>
>>> Log level
>>> <value name="log_level">notice</value>
>>>
>>> When this amount of php processes exited with SIGSEGV
>>> or SIGBUS ...
>>> <value name="emergency_restart_threshold">10</value>
>>>
>>> ... in a less than this interval of time, a graceful
>>> restart will be initiated.
>>> Useful to work around accidental curruptions in
>>> accelerator's shared memory.
>>> <value name="emergency_restart_interval">1m</value>
>>>
>>> Time limit on waiting child's reaction on signals from
>>> master
>>> <value name="process_control_timeout">5s</value>
>>>
>>> Set to 'no' to debug fpm
>>> <value name="daemonize">yes</value>
>>>
>>> </section>
>>>
>>> <workers>
>>>
>>> <section name="pool">
>>>
>>> Name of pool. Used in logs and stats.
>>> <value name="name">nginx</value>
>>>
>>> Address to accept fastcgi requests on.
>>> Valid syntax is 'ip.ad.re.ss:port' or just
>>> 'port' or '/path/to/unix/socket'
>>> <value name="listen_address">127.0.0.1:8000</
>>> value>
>>>
>>> <value name="listen_options">
>>>
>>> Set listen(2) backlog
>>> <value name="backlog">-1</value>
>>>
>>> Set permissions for unix socket, if
>>> one used.
>>> In Linux read/write permissions must
>>> be set in order to allow connections from web server.
>>> Many BSD-derrived systems allow
>>> connections regardless of permissions.
>>> <value name="owner"></value>
>>> <value name="group"></value>
>>> <value name="mode">0666</value>
>>> </value>
>>>
>>> Additional php.ini defines, specific to this
>>> pool of workers.
>>> <value name="php_defines">
>>> 
>>> 
>>> </value>
>>>
>>> Unix user of processes
>>> <value name="user">nobody</value>
>>>
>>> Unix group of processes
>>> <value name="group">nobody</value>
>>>
>>> Process manager settings
>>> <value name="pm">
>>>
>>> Sets style of controling worker
>>> process count.
>>> Valid values are 'static' and 'apache-
>>> like'
>>> <value name="style">static</value>
>>>
>>> Sets the limit on the number of
>>> simultaneous requests that will be served.
>>> Equivalent to Apache MaxClients
>>> directive.
>>> Equivalent to PHP_FCGI_CHILDREN
>>> environment in original php.fcgi
>>> Used with any pm_style.
>>> <value name="max_children">256</value>
>>>
>>> Settings group for 'apache-like' pm
>>> style
>>> <value name="apache_like">
>>>
>>> Sets the number of server
>>> processes created on startup.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value name="StartServers">32</
>>> value>
>>>
>>> Sets the desired minimum
>>> number of idle server processes.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MinSpareServers">64</value>
>>>
>>> Sets the desired maximum
>>> number of idle server processes.
>>> Used only when 'apache-like'
>>> pm_style is selected
>>> <value
>>> name="MaxSpareServers">128</value>
>>>
>>> </value>
>>>
>>> </value>
>>>
>>> The timeout (in seconds) for serving a single
>>> request after which the worker process will be terminated
>>> Should be used when 'max_execution_time' ini
>>> option does not stop script execution for some reason
>>> '0s' means 'off'
>>> <value name="request_terminate_timeout">0s</
>>> value>
>>>
>>> The timeout (in seconds) for serving of single
>>> request after which a php backtrace will be dumped to slow.log file
>>> '0s' means 'off'
>>> <value name="request_slowlog_timeout">10s</
>>> value>
>>>
>>> The log file for slow requests
>>> <value name="slowlog">logs/slow.log</value>
>>>
>>> Set open file desc rlimit
>>> <value name="rlimit_files">8092</value>
>>>
>>> Set max core size rlimit
>>> <value name="rlimit_core">0</value>
>>>
>>> Chroot to this directory at the start,
>>> absolute path
>>> <value name="chroot"></value>
>>>
>>> Chdir to this directory at the start, absolute
>>> path
>>> <value name="chdir"></value>
>>>
>>> Redirect workers' stdout and stderr into main
>>> error log.
>>> If not set, they will be redirected to /dev/
>>> null, according to FastCGI specs
>>> <value name="catch_workers_output">yes</value>
>>>
>>> How much requests each process should execute
>>> before respawn.
>>> Useful to work around memory leaks in 3rd
>>> party libraries.
>>> For endless request processing please specify
>>> 0
>>> Equivalent to PHP_FCGI_MAX_REQUESTS
>>> <value name="max_requests">500</value>
>>>
>>> Comma separated list of ipv4 addresses of
>>> FastCGI clients that allowed to connect.
>>> Equivalent to FCGI_WEB_SERVER_ADDRS
>>> environment in original php.fcgi (5.2.2+)
>>> Makes sense only with AF_INET listening
>>> socket.
>>> <value name="allowed_clients">127.0.0.1</
>>> value>
>>>
>>> Pass environment variables like
>>> LD_LIBRARY_PATH
>>> All $VARIABLEs are taken from current
>>> environment
>>> <value name="environment">
>>> <value name="HOSTNAME">$HOSTNAME</
>>> value>
>>> <value name="PATH">/usr/local/bin:/usr/
>>> bin:/bin</value>
>>> <value name="TMP">/tmp</value>
>>> <value name="TMPDIR">/tmp</value>
>>> <value name="TEMP">/tmp</value>
>>> <value name="OSTYPE">$OSTYPE</value>
>>> <value name="MACHTYPE">$MACHTYPE</
>>> value>
>>> <value name="MALLOC_CHECK_">2</value>
>>> </value>
>>>
>>> </section>
>>>
>>> </workers>
>>>
>>> </configuration>
>>>
>>> 3 - the relevant portion of nginx config:
>>>
>>> user nobody;
>>> worker_processes 64;
>>> worker_rlimit_nofile 10240;
>>>
>>> error_log logs/error.log;
>>> pid logs/nginx.pid;
>>>
>>> events {
>>> worker_connections 8192;
>>> use kqueue;
>>> }
>>>
>>> [...............................]
>>>
>>> location ~* \.php$ {
>>> fastcgi_pass 127.0.0.1:8000;
>>> fastcgi_index index.php;
>>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>>> public_html$fastcgi_script_name;
>>> fastcgi_param SCRIPT_NAME
>>> $fastcgi_script_name;
>>> fastcgi_param REQUEST_URI $request_uri;
>>> fastcgi_param DOCUMENT_URI $document_uri;
>>> fastcgi_param DOCUMENT_ROOT $document_root;
>>> fastcgi_param QUERY_STRING $query_string;
>>> fastcgi_param REQUEST_METHOD $request_method;
>>> fastcgi_param CONTENT_TYPE $content_type;
>>> fastcgi_param CONTENT_LENGTH $content_length;
>>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>>> fastcgi_param SERVER_SOFTWARE nginx;
>>> fastcgi_param REMOTE_ADDR $remote_addr;
>>> fastcgi_param REMOTE_PORT $remote_port;
>>> fastcgi_param SERVER_ADDR $server_addr;
>>> fastcgi_param SERVER_PORT $server_port;
>>> fastcgi_param SERVER_NAME $server_name;
>>> fastcgi_param SERVER_PROTOCOL
>>> $server_protocol;
>>>
>>> fastcgi_buffers 64 64k;
>>> fastcgi_buffer_size 64k;
>>>
>>
>>
>>
>
>

Reply Quote

iberkner

Re: Nginx high load - php stops at some point
September 28, 2010 03:50PM

Registered: 15 years ago
Posts: 156

Some more information regarding mysql:

1. DNS resolution, especially if you're connecting from more than 1 system.
msyql attemps to resolve every connection host (forward and backwards) and
if for some reason it fails, forget it. You're in lala land. We had this
issue a few weeks back with a new application server, forgot to add it to
the hosts file of the db. One option is to turn off DNS resolution, using
--skip-name-resolve option when starting.

2. depending on which version of mysql you're running, slow query log time
parameter has to be integer, i.e. 1 second, 2 seconds, etc. Newer versions
and some percona high performance version allow you to do decimals, for
example, we set ours to 0.5 to catch anything that takes longer than half a
second.

3. Try to understand / profiler your PHP pages, how many queries are you
running per page? how long does it take? are your queries optimized to use
indexes?

I know that the above is not directly related to php-fpm but we've had a lot
of success by tunning mysql AND adding more application servers (have 4 in
rotation right now).

On Tue, Sep 28, 2010 at 3:02 PM, Jonathan Langevin <intel352@gmail.com>wrote:

> Considering the connection limit issue in MySQL, it's also possible your
> MySQL connections are not terminating.
> You can watch this through "mytop" (similar to top/htop, but for MySQL
> query processes).
>
> On siege, you can limit the amount of time to request (I forget the
> parameter). This can help you with benchmarking, so that you can test within
> a 1 minute benchmark, and keep increasing number of concurrent connections.
> Then you also get a nice report summary from Siege, telling you the
> availability % of your requests, request time, concurrency avg, etc.
> Keep in mind there are limitations on concurrency within certain operating
> systems, both outgoing and incoming.
>
> On Tue, Sep 28, 2010 at 2:56 PM, StefanitaRares Dumitrescu <
> themediaplanet90210@gmail.com> wrote:
>
>> Actually i think i modified my config at some point and took out slow log
>> from sql. I need to fix this now. Will see where i go with it.
>>
>> Regarding the connection limit for SQL - i have it at 2000. Should be more
>> than enough imho.
>>
>>
>> On 9/28/2010 7:28 PM, Ilan Berkner wrote:
>>
>> We run a very highload site using Nginx and PHP 5.3.3 (with FPM) with
>> eAccelerator.
>>
>> We are consistently at 5000+ connections, during peak, we've hit 7,000+.
>>
>>
>> I'm always finding stability issues due to the high-load but usually can
>> locate areas in the code base which cause PHP to hang or wait. In
>> particular, check database queries. We've had to separate our data requests
>> into a master->read only slave configuration.
>>
>> Do you have your slow query log turned on and have you looked at what's
>> in there?
>>
>> We're also going to be caching a lot of database calls. The idea is to
>> reduce the load and wait time on the PHP app servers.
>>
>> On Tue, Sep 28, 2010 at 1:21 PM, StefanitaRares Dumitrescu <
>> themediaplanet90210@gmail.com> wrote:
>>
>>> I tried APC prior to eAccelerator, and i had the same issues. It
>>> actually happened more often for php to hang than now with eaccelerator.
>>>
>>> On 9/28/2010 7:19 PM, Jonathan Langevin wrote:
>>>
>>> I was experiencing similar issues recently.
>>> Changed my opcode cache to APC, haven't seen the same problem since.
>>>
>>> Try APC and see if you still have the same issues.
>>>
>>> -Jon
>>>
>>> On Tue, Sep 28, 2010 at 1:15 PM, katmai <themediaplanet90210@gmail.com>wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I am running a very busy website. It's running a photo gallery ->
>>>> Gallery2.
>>>>
>>>> The system is a Quad core Xeon, with 16 gb RAM, and separate drives
>>>> for OS/SQL/site, running FreeBSD 7.2 - latest patch level.
>>>>
>>>> At some point during the day, pretty much random, the load on the
>>>> server spikes, and then php processing stops. I run nginx 0.8.52, and
>>>> php 5.2.10 with fpm patch, and i am using eAccelerator as an opcode
>>>> cache.
>>>>
>>>> I have been debugging and i noticed that when the server reaches over
>>>> 6000 active connections, then everything gets clogged and eventually i
>>>> had to build a script that would restart php/nginx when the load
>>>> starts spiking, which is not quite a nice solution. I don't like doing
>>>> patchwork like this, but i had to.
>>>>
>>>> The behavior i noticed was when the php started dieing, then all php
>>>> processes would be in "pfault" or "sbwait" state.
>>>>
>>>> If you need any more infos i will be happy to provide. i just want to
>>>> know what may be the cause of this.
>>>>
>>>> Information:
>>>> 1 - This is what nginx_status page shows when php is close to dieing.
>>>>
>>>> Active connections: 5018
>>>> server accepts handled requests
>>>> 11836 11836 28147
>>>> Reading: 49 Writing: 2103 Waiting: 2866
>>>>
>>>> 2 - my php-fpm.conf
>>>>
>>>>
>>>> <?xml version="1.0" ?>
>>>> <configuration>
>>>>
>>>> All relative paths in this config are relative to php's
>>>> install prefix
>>>>
>>>> <section name="global_options">
>>>>
>>>> Pid file
>>>> <value name="pid_file">/usr/local/kphp/logs/php-
>>>> fpm.pid</value>
>>>>
>>>> Error log file
>>>> <value name="error_log">/usr/local/kphp/logs/php-
>>>> fpm.log</value>
>>>>
>>>> Log level
>>>> <value name="log_level">notice</value>
>>>>
>>>> When this amount of php processes exited with SIGSEGV
>>>> or SIGBUS ...
>>>> <value name="emergency_restart_threshold">10</value>
>>>>
>>>> ... in a less than this interval of time, a graceful
>>>> restart will be initiated.
>>>> Useful to work around accidental curruptions in
>>>> accelerator's shared memory.
>>>> <value name="emergency_restart_interval">1m</value>
>>>>
>>>> Time limit on waiting child's reaction on signals from
>>>> master
>>>> <value name="process_control_timeout">5s</value>
>>>>
>>>> Set to 'no' to debug fpm
>>>> <value name="daemonize">yes</value>
>>>>
>>>> </section>
>>>>
>>>> <workers>
>>>>
>>>> <section name="pool">
>>>>
>>>> Name of pool. Used in logs and stats.
>>>> <value name="name">nginx</value>
>>>>
>>>> Address to accept fastcgi requests on.
>>>> Valid syntax is 'ip.ad.re.ss:port' or just
>>>> 'port' or '/path/to/unix/socket'
>>>> <value name="listen_address">127.0.0.1:8000</
>>>> value>
>>>>
>>>> <value name="listen_options">
>>>>
>>>> Set listen(2) backlog
>>>> <value name="backlog">-1</value>
>>>>
>>>> Set permissions for unix socket, if
>>>> one used.
>>>> In Linux read/write permissions must
>>>> be set in order to allow connections from web server.
>>>> Many BSD-derrived systems allow
>>>> connections regardless of permissions.
>>>> <value name="owner"></value>
>>>> <value name="group"></value>
>>>> <value name="mode">0666</value>
>>>> </value>
>>>>
>>>> Additional php.ini defines, specific to this
>>>> pool of workers.
>>>> <value name="php_defines">
>>>> 
>>>> 
>>>> </value>
>>>>
>>>> Unix user of processes
>>>> <value name="user">nobody</value>
>>>>
>>>> Unix group of processes
>>>> <value name="group">nobody</value>
>>>>
>>>> Process manager settings
>>>> <value name="pm">
>>>>
>>>> Sets style of controling worker
>>>> process count.
>>>> Valid values are 'static' and 'apache-
>>>> like'
>>>> <value name="style">static</value>
>>>>
>>>> Sets the limit on the number of
>>>> simultaneous requests that will be served.
>>>> Equivalent to Apache MaxClients
>>>> directive.
>>>> Equivalent to PHP_FCGI_CHILDREN
>>>> environment in original php.fcgi
>>>> Used with any pm_style.
>>>> <value name="max_children">256</value>
>>>>
>>>> Settings group for 'apache-like' pm
>>>> style
>>>> <value name="apache_like">
>>>>
>>>> Sets the number of server
>>>> processes created on startup.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value name="StartServers">32</
>>>> value>
>>>>
>>>> Sets the desired minimum
>>>> number of idle server processes.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MinSpareServers">64</value>
>>>>
>>>> Sets the desired maximum
>>>> number of idle server processes.
>>>> Used only when 'apache-like'
>>>> pm_style is selected
>>>> <value
>>>> name="MaxSpareServers">128</value>
>>>>
>>>> </value>
>>>>
>>>> </value>
>>>>
>>>> The timeout (in seconds) for serving a single
>>>> request after which the worker process will be terminated
>>>> Should be used when 'max_execution_time' ini
>>>> option does not stop script execution for some reason
>>>> '0s' means 'off'
>>>> <value name="request_terminate_timeout">0s</
>>>> value>
>>>>
>>>> The timeout (in seconds) for serving of single
>>>> request after which a php backtrace will be dumped to slow.log file
>>>> '0s' means 'off'
>>>> <value name="request_slowlog_timeout">10s</
>>>> value>
>>>>
>>>> The log file for slow requests
>>>> <value name="slowlog">logs/slow.log</value>
>>>>
>>>> Set open file desc rlimit
>>>> <value name="rlimit_files">8092</value>
>>>>
>>>> Set max core size rlimit
>>>> <value name="rlimit_core">0</value>
>>>>
>>>> Chroot to this directory at the start,
>>>> absolute path
>>>> <value name="chroot"></value>
>>>>
>>>> Chdir to this directory at the start, absolute
>>>> path
>>>> <value name="chdir"></value>
>>>>
>>>> Redirect workers' stdout and stderr into main
>>>> error log.
>>>> If not set, they will be redirected to /dev/
>>>> null, according to FastCGI specs
>>>> <value name="catch_workers_output">yes</value>
>>>>
>>>> How much requests each process should execute
>>>> before respawn.
>>>> Useful to work around memory leaks in 3rd
>>>> party libraries.
>>>> For endless request processing please specify
>>>> 0
>>>> Equivalent to PHP_FCGI_MAX_REQUESTS
>>>> <value name="max_requests">500</value>
>>>>
>>>> Comma separated list of ipv4 addresses of
>>>> FastCGI clients that allowed to connect.
>>>> Equivalent to FCGI_WEB_SERVER_ADDRS
>>>> environment in original php.fcgi (5.2.2+)
>>>> Makes sense only with AF_INET listening
>>>> socket.
>>>> <value name="allowed_clients">127.0.0.1</
>>>> value>
>>>>
>>>> Pass environment variables like
>>>> LD_LIBRARY_PATH
>>>> All $VARIABLEs are taken from current
>>>> environment
>>>> <value name="environment">
>>>> <value name="HOSTNAME">$HOSTNAME</
>>>> value>
>>>> <value name="PATH">/usr/local/bin:/usr/
>>>> bin:/bin</value>
>>>> <value name="TMP">/tmp</value>
>>>> <value name="TMPDIR">/tmp</value>
>>>> <value name="TEMP">/tmp</value>
>>>> <value name="OSTYPE">$OSTYPE</value>
>>>> <value name="MACHTYPE">$MACHTYPE</
>>>> value>
>>>> <value name="MALLOC_CHECK_">2</value>
>>>> </value>
>>>>
>>>> </section>
>>>>
>>>> </workers>
>>>>
>>>> </configuration>
>>>>
>>>> 3 - the relevant portion of nginx config:
>>>>
>>>> user nobody;
>>>> worker_processes 64;
>>>> worker_rlimit_nofile 10240;
>>>>
>>>> error_log logs/error.log;
>>>> pid logs/nginx.pid;
>>>>
>>>> events {
>>>> worker_connections 8192;
>>>> use kqueue;
>>>> }
>>>>
>>>> [...............................]
>>>>
>>>> location ~* \.php$ {
>>>> fastcgi_pass 127.0.0.1:8000;
>>>> fastcgi_index index.php;
>>>> fastcgi_param SCRIPT_FILENAME /xdrive/cityv3/
>>>> public_html$fastcgi_script_name;
>>>> fastcgi_param SCRIPT_NAME
>>>> $fastcgi_script_name;
>>>> fastcgi_param REQUEST_URI $request_uri;
>>>> fastcgi_param DOCUMENT_URI $document_uri;
>>>> fastcgi_param DOCUMENT_ROOT $document_root;
>>>> fastcgi_param QUERY_STRING $query_string;
>>>> fastcgi_param REQUEST_METHOD $request_method;
>>>> fastcgi_param CONTENT_TYPE $content_type;
>>>> fastcgi_param CONTENT_LENGTH $content_length;
>>>> fastcgi_param GATEWAY_INTERFACE CGI/1.1;
>>>> fastcgi_param SERVER_SOFTWARE nginx;
>>>> fastcgi_param REMOTE_ADDR $remote_addr;
>>>> fastcgi_param REMOTE_PORT $remote_port;
>>>> fastcgi_param SERVER_ADDR $server_addr;
>>>> fastcgi_param SERVER_PORT $server_port;
>>>> fastcgi_param SERVER_NAME $server_name;
>>>> fastcgi_param SERVER_PROTOCOL
>>>> $server_protocol;
>>>>
>>>> fastcgi_buffers 64 64k;
>>>> fastcgi_buffer_size 64k;
>>>>
>>>
>>>
>>>
>>
>>
>

Reply Quote

Maciej Lisiewski

Re: Nginx high load - php stops at some point
September 28, 2010 07:46PM

> Regarding the connection limit for SQL - i have it at 2000. Should be
> more than enough imho.

This one you can actually easily check - just run this query in MySQL:
SHOW STATUS LIKE 'Max_used_connections';

--
Maciej Lisiewski

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 29, 2010 09:40AM

happened today again. once the server goes over 6000 active
connections, nginx goes into pfault state, and php into sbwait.
everything times out on the site.

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
55524 nobody 1 105 0 2275M 1766M pfault 4 5:21 15.33% nginx
55517 nobody 1 105 0 1909M 1465M pfault 6 5:18 15.19% nginx
55525 nobody 1 105 0 1831M 1336M pfault 3 5:06 15.09% nginx
55522 nobody 1 105 0 1928M 1379M pfault 0 5:20 14.65% nginx
55521 nobody 1 105 0 2132M 1658M pfault 4 5:15 14.21% nginx
55523 nobody 1 104 0 1998M 1386M pfault 7 5:13 13.92% nginx
55518 nobody 1 104 0 2142M 1419M pfault 7 5:28 13.62% nginx
55526 nobody 1 104 0 1839M 1294M pfault 7 5:11 13.23% nginx
56345 nobody 1 4 0 76140K 16284K sbwait 4 0:04 2.60% php-cgi
56218 nobody 1 4 0 76140K 15228K sbwait 0 0:04 2.39% php-cgi
56261 nobody 1 4 0 76140K 18632K sbwait 5 0:04 1.96% php-cgi
56244 nobody 1 97 0 85356K 23712K pfault 6 0:03 1.96% php-cgi
56516 nagios 1 8 0 9616K 1740K wait 5 0:00 1.53% nrpe
3794 mysql 130 4 0 1219M 365M sbwait 1 0:16 1.42% mysqld
56205 nobody 1 4 0 76140K 5384K sbwait 2 0:03 1.24% php-cgi
56344 nobody 1 4 0 76140K 20080K sbwait 7 0:03 1.16% php-cgi
56416 nobody 1 4 0 76140K 19672K sbwait 5 0:02 1.05% php-cgi
56368 nobody 1 4 0 76140K 16168K sbwait 5 0:03 1.03% php-cgi
56365 nobody 1 4 0 76140K 8588K sbwait 6 0:03 1.00% php-cgi
56204 nobody 1 4 0 76140K 15464K sbwait 6 0:02 0.99% php-cgi
56372 nobody 1 4 0 76140K 18456K sbwait 6 0:04 0.95% php-cgi
56405 nobody 1 4 0 77164K 14344K sbwait 7 0:04 0.78% php-cgi
56322 nobody 1 4 0 76140K 16368K sbwait 3 0:03 0.78% php-cgi
56347 nobody 1 4 0 76140K 17664K sbwait 6 0:03 0.77% php-cgi
56436 nobody 1 4 0 76140K 17236K sbwait 1 0:03 0.73% php-cgi
56389 nobody 1 4 0 76140K 21092K sbwait 7 0:03 0.73% php-cgi
56269 nobody 1 4 0 76140K 12852K sbwait 3 0:03 0.72% php-cgi
56246 nobody 1 4 0 76140K 14984K sbwait 7 0:03 0.72% php-cgi

On 9/29/2010 1:45 AM, Maciej Lisiewski wrote:
>> Regarding the connection limit for SQL - i have it at 2000. Should be
>> more than enough imho.
>
> This one you can actually easily check - just run this query in MySQL:
> SHOW STATUS LIKE 'Max_used_connections';
>
>

Reply Quote

Jérôme Loyet

Re: Nginx high load - php stops at some point
September 29, 2010 10:20AM

you should monitor the number of active connections to your database.

2010/9/29 StefanitaRares Dumitrescu <themediaplanet90210@gmail.com>:
> happened today again. once the server goes over 6000 active connections,
> nginx goes into pfault state, and php into sbwait. everything times out on
> the site.
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 55524 nobody 1 105 0 2275M 1766M pfault 4 5:21 15.33% nginx
> 55517 nobody 1 105 0 1909M 1465M pfault 6 5:18 15.19% nginx
> 55525 nobody 1 105 0 1831M 1336M pfault 3 5:06 15.09% nginx
> 55522 nobody 1 105 0 1928M 1379M pfault 0 5:20 14.65% nginx
> 55521 nobody 1 105 0 2132M 1658M pfault 4 5:15 14.21% nginx
> 55523 nobody 1 104 0 1998M 1386M pfault 7 5:13 13.92% nginx
> 55518 nobody 1 104 0 2142M 1419M pfault 7 5:28 13.62% nginx
> 55526 nobody 1 104 0 1839M 1294M pfault 7 5:11 13.23% nginx
> 56345 nobody 1 4 0 76140K 16284K sbwait 4 0:04 2.60% php-cgi
> 56218 nobody 1 4 0 76140K 15228K sbwait 0 0:04 2.39% php-cgi
> 56261 nobody 1 4 0 76140K 18632K sbwait 5 0:04 1.96% php-cgi
> 56244 nobody 1 97 0 85356K 23712K pfault 6 0:03 1.96% php-cgi
> 56516 nagios 1 8 0 9616K 1740K wait 5 0:00 1.53% nrpe
> 3794 mysql 130 4 0 1219M 365M sbwait 1 0:16 1.42% mysqld
> 56205 nobody 1 4 0 76140K 5384K sbwait 2 0:03 1.24% php-cgi
> 56344 nobody 1 4 0 76140K 20080K sbwait 7 0:03 1.16% php-cgi
> 56416 nobody 1 4 0 76140K 19672K sbwait 5 0:02 1.05% php-cgi
> 56368 nobody 1 4 0 76140K 16168K sbwait 5 0:03 1.03% php-cgi
> 56365 nobody 1 4 0 76140K 8588K sbwait 6 0:03 1.00% php-cgi
> 56204 nobody 1 4 0 76140K 15464K sbwait 6 0:02 0.99% php-cgi
> 56372 nobody 1 4 0 76140K 18456K sbwait 6 0:04 0.95% php-cgi
> 56405 nobody 1 4 0 77164K 14344K sbwait 7 0:04 0.78% php-cgi
> 56322 nobody 1 4 0 76140K 16368K sbwait 3 0:03 0.78% php-cgi
> 56347 nobody 1 4 0 76140K 17664K sbwait 6 0:03 0.77% php-cgi
> 56436 nobody 1 4 0 76140K 17236K sbwait 1 0:03 0.73% php-cgi
> 56389 nobody 1 4 0 76140K 21092K sbwait 7 0:03 0.73% php-cgi
> 56269 nobody 1 4 0 76140K 12852K sbwait 3 0:03 0.72% php-cgi
> 56246 nobody 1 4 0 76140K 14984K sbwait 7 0:03 0.72% php-cgi
>
>
> On 9/29/2010 1:45 AM, Maciej Lisiewski wrote:
>>>
>>> Regarding the connection limit for SQL - i have it at 2000. Should be
>>> more than enough imho.
>>
>> This one you can actually easily check - just run this query in MySQL:
>> SHOW STATUS LIKE 'Max_used_connections';
>>
>>
>
>

Reply Quote

iberkner

Re: Nginx high load - php stops at some point
September 29, 2010 10:30AM

Registered: 15 years ago
Posts: 156

You could profile your PHP application using xdebug (note it puts a load on
the system of course), for a brief period and review the generated grind
files to identify where php is getting stuck.

2010/9/29 Jérôme Loyet <ml@fatbsd.com>

> you should monitor the number of active connections to your database.
>
> 2010/9/29 StefanitaRares Dumitrescu <themediaplanet90210@gmail.com>:
> > happened today again. once the server goes over 6000 active connections,
> > nginx goes into pfault state, and php into sbwait. everything times out
> on
> > the site.
> >
> > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> > 55524 nobody 1 105 0 2275M 1766M pfault 4 5:21 15.33% nginx
> > 55517 nobody 1 105 0 1909M 1465M pfault 6 5:18 15.19% nginx
> > 55525 nobody 1 105 0 1831M 1336M pfault 3 5:06 15.09% nginx
> > 55522 nobody 1 105 0 1928M 1379M pfault 0 5:20 14.65% nginx
> > 55521 nobody 1 105 0 2132M 1658M pfault 4 5:15 14.21% nginx
> > 55523 nobody 1 104 0 1998M 1386M pfault 7 5:13 13.92% nginx
> > 55518 nobody 1 104 0 2142M 1419M pfault 7 5:28 13.62% nginx
> > 55526 nobody 1 104 0 1839M 1294M pfault 7 5:11 13.23% nginx
> > 56345 nobody 1 4 0 76140K 16284K sbwait 4 0:04 2.60% php-cgi
> > 56218 nobody 1 4 0 76140K 15228K sbwait 0 0:04 2.39% php-cgi
> > 56261 nobody 1 4 0 76140K 18632K sbwait 5 0:04 1.96% php-cgi
> > 56244 nobody 1 97 0 85356K 23712K pfault 6 0:03 1.96% php-cgi
> > 56516 nagios 1 8 0 9616K 1740K wait 5 0:00 1.53% nrpe
> > 3794 mysql 130 4 0 1219M 365M sbwait 1 0:16 1.42% mysqld
> > 56205 nobody 1 4 0 76140K 5384K sbwait 2 0:03 1.24% php-cgi
> > 56344 nobody 1 4 0 76140K 20080K sbwait 7 0:03 1.16% php-cgi
> > 56416 nobody 1 4 0 76140K 19672K sbwait 5 0:02 1.05% php-cgi
> > 56368 nobody 1 4 0 76140K 16168K sbwait 5 0:03 1.03% php-cgi
> > 56365 nobody 1 4 0 76140K 8588K sbwait 6 0:03 1.00% php-cgi
> > 56204 nobody 1 4 0 76140K 15464K sbwait 6 0:02 0.99% php-cgi
> > 56372 nobody 1 4 0 76140K 18456K sbwait 6 0:04 0.95% php-cgi
> > 56405 nobody 1 4 0 77164K 14344K sbwait 7 0:04 0.78% php-cgi
> > 56322 nobody 1 4 0 76140K 16368K sbwait 3 0:03 0.78% php-cgi
> > 56347 nobody 1 4 0 76140K 17664K sbwait 6 0:03 0.77% php-cgi
> > 56436 nobody 1 4 0 76140K 17236K sbwait 1 0:03 0.73% php-cgi
> > 56389 nobody 1 4 0 76140K 21092K sbwait 7 0:03 0.73% php-cgi
> > 56269 nobody 1 4 0 76140K 12852K sbwait 3 0:03 0.72% php-cgi
> > 56246 nobody 1 4 0 76140K 14984K sbwait 7 0:03 0.72% php-cgi
> >
> >
> > On 9/29/2010 1:45 AM, Maciej Lisiewski wrote:
> >>>
> >>> Regarding the connection limit for SQL - i have it at 2000. Should be
> >>> more than enough imho.
> >>
> >> This one you can actually easily check - just run this query in MySQL:
> >> SHOW STATUS LIKE 'Max_used_connections';
> >>
> >>
> >
> >
>

Reply Quote

StefanitaRares Dumitrescu

Re: Nginx high load - php stops at some point
September 29, 2010 10:50AM

It's not an sql problem imho. the maximum number of used connections
over the past 19 hours is 254. also sql has it's own drive, so the i/o
is not shared with the os or files.

Historic max_used_connections = 254

the drive where the website resides, is a ssd drive, however when the
spikes occur, the i/o shoots through the roof. the i/o is in Mb/s - last
column represents the i/o on the website drive ... 39mb/s / 31.84 and so on.

21.15 206 4.25 25.11 158 3.88 32.10 109 3.41
16.00 2 0.03 20.96 238 4.88 31.75 995 30.86
56.90 62 3.44 21.00 179 3.68 29.47 1116 32.13
19.88 16 0.32 16.74 183 3.00 32.19 1250 39.29
21.20 15 0.31 18.94 202 3.74 32.39 1007 31.84
22.20 10 0.22 24.90 200 4.87 29.38 870 24.95
8.64 12 0.11 25.15 250 6.14 33.57 796 26.08
26.00 3 0.08 21.33 204 4.25 40.26 798 31.38
26.74 9 0.25 19.71 177 3.41 41.55 757 30.70

nginx_status shows that when going over 6500 active connections
everything starts failing.

Active connections: 6055
server accepts handled requests
10477 10477 19521
Reading: 127 Writing: 3751 Waiting: 2177

On 9/29/2010 4:14 PM, Jérôme Loyet wrote:
> you should monitor the number of active connections to your database.
>
> 2010/9/29 StefanitaRares Dumitrescu<themediaplanet90210@gmail.com>:
>> happened today again. once the server goes over 6000 active connections,
>> nginx goes into pfault state, and php into sbwait. everything times out on
>> the site.
>>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>> 55524 nobody 1 105 0 2275M 1766M pfault 4 5:21 15.33% nginx
>> 55517 nobody 1 105 0 1909M 1465M pfault 6 5:18 15.19% nginx
>> 55525 nobody 1 105 0 1831M 1336M pfault 3 5:06 15.09% nginx
>> 55522 nobody 1 105 0 1928M 1379M pfault 0 5:20 14.65% nginx
>> 55521 nobody 1 105 0 2132M 1658M pfault 4 5:15 14.21% nginx
>> 55523 nobody 1 104 0 1998M 1386M pfault 7 5:13 13.92% nginx
>> 55518 nobody 1 104 0 2142M 1419M pfault 7 5:28 13.62% nginx
>> 55526 nobody 1 104 0 1839M 1294M pfault 7 5:11 13.23% nginx
>> 56345 nobody 1 4 0 76140K 16284K sbwait 4 0:04 2.60% php-cgi
>> 56218 nobody 1 4 0 76140K 15228K sbwait 0 0:04 2.39% php-cgi
>> 56261 nobody 1 4 0 76140K 18632K sbwait 5 0:04 1.96% php-cgi
>> 56244 nobody 1 97 0 85356K 23712K pfault 6 0:03 1.96% php-cgi
>> 56516 nagios 1 8 0 9616K 1740K wait 5 0:00 1.53% nrpe
>> 3794 mysql 130 4 0 1219M 365M sbwait 1 0:16 1.42% mysqld
>> 56205 nobody 1 4 0 76140K 5384K sbwait 2 0:03 1.24% php-cgi
>> 56344 nobody 1 4 0 76140K 20080K sbwait 7 0:03 1.16% php-cgi
>> 56416 nobody 1 4 0 76140K 19672K sbwait 5 0:02 1.05% php-cgi
>> 56368 nobody 1 4 0 76140K 16168K sbwait 5 0:03 1.03% php-cgi
>> 56365 nobody 1 4 0 76140K 8588K sbwait 6 0:03 1.00% php-cgi
>> 56204 nobody 1 4 0 76140K 15464K sbwait 6 0:02 0.99% php-cgi
>> 56372 nobody 1 4 0 76140K 18456K sbwait 6 0:04 0.95% php-cgi
>> 56405 nobody 1 4 0 77164K 14344K sbwait 7 0:04 0.78% php-cgi
>> 56322 nobody 1 4 0 76140K 16368K sbwait 3 0:03 0.78% php-cgi
>> 56347 nobody 1 4 0 76140K 17664K sbwait 6 0:03 0.77% php-cgi
>> 56436 nobody 1 4 0 76140K 17236K sbwait 1 0:03 0.73% php-cgi
>> 56389 nobody 1 4 0 76140K 21092K sbwait 7 0:03 0.73% php-cgi
>> 56269 nobody 1 4 0 76140K 12852K sbwait 3 0:03 0.72% php-cgi
>> 56246 nobody 1 4 0 76140K 14984K sbwait 7 0:03 0.72% php-cgi
>>
>>
>> On 9/29/2010 1:45 AM, Maciej Lisiewski wrote:
>>>> Regarding the connection limit for SQL - i have it at 2000. Should be
>>>> more than enough imho.
>>> This one you can actually easily check - just run this query in MySQL:
>>> SHOW STATUS LIKE 'Max_used_connections';
>>>
>>>
>>

Reply Quote

Nginx high load - php stops at some point

Online Users