I was looking everywhere, but I could never find a good guide that explains how to configure Nginx with PHP-FPM for a heavy server load, so I hope this is the place where I will find my answer.

I have a 24 core server with 32GB RAM with Centos 5.6 which is running Nginx 1.0.2 web server with PHP-FPM (PHP 5.3.5) which is hosting a website that is running some very easy and fast php scripts which are mostly making calls to external sites. The server is not on a heavy load at all, the CPU is never above 25% (usually around 5-10%) and there is 24GB RAM free. The website is getting around 130 hits per seconds and even more during the peaks of the day.

In the Nginx error log most of the errors are:
upstream timed out (110: Connection timed out) while reading response header from upstream

and not that often:

connect() to unix:/dev/shm/.php-fpm/socket failed (11: Resource temporarily unavailable) while connecting to upstream

The error log can get filled to more than 1MB per hour sometimes, and sometimes we get less of those errors, but they are always there.

Here is the current nginx.conf:
-------------------------------------------------------
user nginx;
worker_processes 8;
worker_rlimit_nofile 65536;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events
{
worker_connections 10240;
use epoll;
}


http
{
include mime.types;
default_type application/octet-stream;

access_log off;
sendfile on;
keepalive_timeout 30;
tcp_nodelay on;
server_tokens off;

gzip on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
gzip_buffers 16 8k;

upstream backend {
server unix:/dev/shm/.php-fpm/socket;
server unix:/dev/shm/.php-fpm/socket2;
}

server
{
listen 80 default;
root /www/php;

location ~* ^.+.(jpg|jpeg|gif|png|css|js|ico|swf|xml|htm|html)$
{
expires 30d;
access_log off;
}

location /
{
index index.html index.php;
}

location ~ \.php$
{
include fastcgi_params;
fastcgi_pass backend;
fastcgi_index index.php;
fastcgi_send_timeout 15;
fastcgi_read_timeout 15;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}

location ~ /\.ht
{
deny all;
}
}
}
-------------------------------------------------------

and php-fpm.conf:
-------------------------------------------------------
[global]
pid = /var/run/php-fpm.pid
error_log = /var/log/php-fpm/php-fpm.log
log_level = error
daemonize = yes

[www]
listen = /dev/shm/.php-fpm/socket
user = php
group = php
pm = dynamic
pm.max_children = 4096
pm.start_servers = 20
pm.min_spare_servers = 5
pm.max_spare_servers = 128
pm.max_requests = 4096
pm.status_path = /status

[www2]
listen = /dev/shm/.php-fpm/socket2
user = php
group = php
pm = dynamic
pm.max_children = 4098
pm.start_servers = 20
pm.min_spare_servers = 5
pm.max_spare_servers = 128
pm.max_requests = 4096
pm.status_path = /status
-------------------------------------------------------

The reason why we are running two pools is because we had a lot of "11: Resource temporarily unavailable" errors in the nginx log because of the socket acces, so we added another pool so that now there are 2 sockets. I did a few system tweaks to solve this issue (which I'm not still sure if it's solved) by increasing the backlog and somaxconn:

net.core.netdev_max_backlog = 4096
net.core.somaxconn = 4096

PHP-FPM log is always empty and I when I look at the status of php-fpm I get something like this:
pool www
process manager dynamic
accepted conn 1111392
listen queue len 0
max listen queue len -1
idle processes 44
active processes 146
total processes 190
max children reached 0

Does anyone have any ideas what could be causing these errors and what settings could I change/tweak to remove as many of them as possible? I'm not sure what is the bottle neck here, is it Nginx (I don't think it is), php-fpm or something else.

Hi,

Answers to your question may be beyond the scope of nginx configuration. For example, there are several sysctl options that control system limits that might affect what you are experiencing. Two of them you have already mentioned. You might want to look for more information about the below options.

Try running the below commands from the same shell where you start nginx, then restart nginx and see if it helps:

# increase the number of maximum open files on your system
sysctl -w fs.file-max=209708

# secure values with reserve for web processes
# Increase number of incoming connections backlog
sysctl -w net.core.netdev_max_backlog=4096
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcp_max_syn_backlog=4096

You may try changing the above values depending on your system. If it works, you may specify them in your /etc/sysctl.conf file.


php-fpm.conf:

# try this per pool:
listen.backlog = 4096

---

nginx options that you should look at in that fine manual out there:
(everything is system-specific, but commented options are even more so)

http {
[...]
client_header_timeout 10m;
client_body_timeout 10m;
send_timeout 10m;

client_max_body_size 700m;
connection_pool_size 256;
client_body_buffer_size 1024k;
client_header_buffer_size 8k;
# large_client_header_buffers 4 8k;
# request_pool_size 4k;

# the below options depend on theoretical maximum of your PHP script run-time
fastcgi_read_timeout 300;
fastcgi_send_timeout 300;


You might want to alter upstream backend options per server:

upstream backend {
server unix:/dev/shm/.php-fpm/socket weight=100 max_fails=3 fail_timeout=5;
server unix:/dev/shm/.php-fpm/socket2 weight=100 max_fails=3 fail_timeout=5;
}


Let me know if any of the suggestions work for you. Don't take the example values provided as fully suitable for your system. You must research and experiment to provide correct values for your configuration.

Good luck!

Andrejs

Thanks for a fast reply I will try to tweak all those settings.

Max number of file descriptors is already very high, so that shouldn't be an issue.

fs.file-max = 3240184

I increased other settings as well in sysctl.conf

I think the main problem was the listen.backlog = 4096 setting, which wasn't set since I thought that the default setting -1 which means that it reads the system backlog settings, but that is not true. The default value is 128.

I will try other suggestions as well and report the results.

This is late, but do you have a solution taht worked?