Добрый день.

Установлена связка nginx/0.8.53 и php-fpm в поставке с php 5.3.3.
Все работает хорошо, за исключением одного момента.

С административной панели одного их сайтов загружаются графические файлы, загрузка происходит при помощи флеш приложения Uplodify, который передает файлы php скрипту.

1. php получает файл, проверяем по $_FILES
2. функцией imagecopyresampled() кропим 3 раза под разные размеры и сохраняем файлы
3. пишем в базу

Так вот файлы маленьких размеров порядка 200-300 К грузятся и обрабатываются без проблем, файлы больших размеров 3-9 Мб, грузятся не всегда, из 5 загружаемых файлов успешно обрабатываются может 2, может 3, могут вообще никто. Ранее была связка nginx+apache, то есть php обрабатывал apache в режиме suexec, и никаких проблем не было, время конвертирования изображения состовляло не более 30 сек. Щас же более 3-х минут. И не всегда понятно почему одно изображение проходит а некоторе нет, вероятно это связано с размером файла, чем больше файл тем больше времени нужно на обработку. Но показатели 30 сек. против 3 минут это порядок в 10 раз.

В логах Nginx я вижу следующее:

2010/10/27 10:56:55 [warn] 78730#0: *16049 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000001, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 10:57:14 [warn] 78730#0: *17596 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000002, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 10:57:32 [warn] 78727#0: *19076 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000003, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 10:57:54 [warn] 78730#0: *21116 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000004, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 11:00:55 [info] 78730#0: *21116 kevent() reported about an closed connection (54: Connection reset by

peer) while sending request to upstream, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", upstream:

"fastcgi://unix:/home/arriva/arriva.arriva.ru.php.sock:", host: "www.arriva.ru"
2010/10/27 11:00:55 [info] 78730#0: *21116 shutdown() failed (54: Connection reset by peer) while sending to

client, client: 91.197.192.14, server: arriva.ru, request: "POST /admin/application/models/upload.php HTTP/1.1",

upstream: "fastcgi://unix:/home/arriva/arriva.arriva.ru.php.sock:", host: "www.arriva.ru"
2010/10/27 11:00:56 [warn] 78726#0: *35688 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000005, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 11:01:30 [warn] 78726#0: *38610 a client request body is buffered to a temporary file

/var/tmp/nginx/client_body_temp/0000000006, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", host: "www.arriva.ru"
2010/10/27 11:04:31 [info] 78726#0: *38610 kevent() reported about an closed connection (54: Connection reset by

peer) while sending request to upstream, client: 91.197.192.14, server: arriva.ru, request: "POST

/admin/application/models/upload.php HTTP/1.1", upstream:

"fastcgi://unix:/home/arriva/arriva.arriva.ru.php.sock:", host: "www.arriva.ru"
2010/10/27 11:04:31 [info] 78726#0: *38610 shutdown() failed (54: Connection reset by peer) while sending to

client, client: 91.197.192.14, server: arriva.ru, request: "POST /admin/application/models/upload.php HTTP/1.1",

upstream: "fastcgi://unix:/home/arriva/arriva.arriva.ru.php.sock:", host: "www.arriva.ru"
---

Видно как происходит разрыв, рвется по таймауту, который вытcавлен в настройках, если обратить внимание то время после загрузки и разрыва составляет 3 минуты. Что логично.
Вот глобальные настройки для nginx:

user apache apache;
worker_processes 5;

error_log logs/error.log debug;

#pid logs/nginx.pid;
worker_rlimit_nofile 8192;


events {
worker_connections 4096;
use kqueue;
}


http {
include mime.types;
default_type application/octet-stream;


log_format main '$remote_addr - $remote_user [$time_local] $host $request "$status" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" "$upstream_response_time" $request_time';

log_format domain '$http_x_forwarded_for - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$remote_addr" $request_time';

log_format ip '$remote_addr';
log_format bytes '$body_bytes_sent';


# GEO ACL
geo $geo {
include worldip.conf;
default RU;
93.92.26.114 RU;
}

## Proxy
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
proxy_buffers 32 4k;
proxy_ignore_client_abort on;


# 27.09.2010
## Size Limits
client_body_buffer_size 256K;
client_header_buffer_size 256K;
client_max_body_size 100M;
large_client_header_buffers 16 16k;
ssi on;
ssi_value_length 1024;

client_header_timeout 140;
client_body_timeout 140;
send_timeout 140;
keepalive_timeout 300 60;
#keepalive_timeout 0;
ignore_invalid_headers on;
keepalive_requests 100;
server_name_in_redirect off;
server_tokens off;

# limit_zone gulag $http_x_forwarded_for 10m;

## TCP options
tcp_nodelay on;
tcp_nopush on;

output_buffers 16 32k;
#output_buffers 1 128k;

reset_timedout_connection on;
open_file_cache max=100000 inactive=40s;
open_file_cache_valid 60s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

access_log logs/access.log main;

sendfile on;

include virtual/*/*.conf;
-------------

Далее настройки для vhost nginx:

####
server {
listen 91.197.192.12:80;
client_max_body_size 100m;
server_name domain.ru www.domain.ru;

set $root /home/domain/domains/domain.ru/public_html;
root $root;
error_page 500 502 503 504 /50x.html;
location = /50x.html { root html; }

#limit_conn gulag 10;

if ($geo != RU)
{
return 444;
}

location / {
log_not_found on;
#if ($loadavg_1m > 30) { return 503; } # do not use with original nginx
root $root;
index index.html index.htm index.php;

try_files $uri $uri/ /index.php;

# if (-e $request_filename) {
# expires max;
# break;
# }


# if ( !-e $request_filename ) {
# rewrite ^(.+) /index.php last;
#
# }




access_log /var/log/httpd/domains/domain.ru.bytes bytes;
access_log /var/log/httpd/domains/domain.ru.log domain;
access_log /usr/local/etc/nginx/logs/access.log main;

}

location ~ /\.ht { deny all; }

location ~* ^/(webmail|squirrelmail|roundcube|phpMyAdmin)/.+\.(bmp|jpg|jpeg|gif|ico|cur|png|css|doc|txt|js|docx|rtf|ppt|pdf|svg)$ {
client_max_body_size 100m;
expires 24h;
root $root;
gzip off;
log_not_found off;
access_log /var/log/httpd/domains/domain.ru.bytes bytes;
access_log /var/log/httpd/domains/domain.ru.log domain;
}

location ~* ^.+\.(css|htc|js|bmp|jp?g|gif|ico|cur|png|doc|avi|txt|doc|rtf|swf|rar|bz2?|gz|zip|mpe?g|ogg|mp3|svg|xml)$ {
client_max_body_size 100m;
expires 24h;
root $root;
gzip off;
access_log /var/log/httpd/domains/domain.ru.bytes bytes;
access_log /var/log/httpd/domains/domain.ru.log domain;
log_not_found off;
}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {

fastcgi_split_path_info ^(.+\.php)(.*)$;
client_max_body_size 100m;
fastcgi_pass unix:/home/domain/domain.domain.ru.php.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /home/domain/domains/domain.ru/public_html$fastcgi_script_name;
include fastcgi_params;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
#added
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
# end added
fastcgi_intercept_errors on;
fastcgi_ignore_client_abort on;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 3000;
fastcgi_read_timeout 300;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;

}

}


###
------------


Далее настройки пула для php-fpm:

;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

; Multiple pools of child processes may be started with different listening
; ports and different management options. The name of the pool will be
; used in logs and stats. There is no limitation on the number of pools which
; FPM can handle. Your system will tell you anyway :)

; Start a new pool named 'www'.
[domain.domain.ru]

; The address on which to accept FastCGI requests.
; Valid syntaxes are:
; 'ip.add.re.ss:port' - to listen on a TCP socket to a specific address on
; a specific port;
; 'port' - to listen on a TCP socket to all addresses on a
; specific port;
; '/path/to/unix/socket' - to listen on a unix socket.
; Note: This value is mandatory.
;listen = 127.0.0.1:9000

listen = /home/domain/domain.domain.ru.php.sock

; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1

; List of ipv4 addresses of FastCGI clients which are allowed to connect.
; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original
; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address
; must be separated by a comma. If this value is left blank, connections will be
; accepted from any ip address.
; Default Value: any
;listen.allowed_clients = 127.0.0.1

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server. Many
; BSD-derived systems allow connections regardless of permissions.
; Default Values: user and group are set as the running user
; mode is set to 0666
listen.owner = arriva
listen.group = apache
listen.mode = 0666

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
; will be used.
user = arriva
group = arriva

; Choose how the process manager will control the number of child processes.
; Possible Values:
; static - a fixed number (pm.max_children) of child processes;
; dynamic - the number of child processes are set dynamically based on the
; following directives:
; pm.max_children - the maximum number of children that can
; be alive at the same time.
; pm.start_servers - the number of children created on startup.
; pm.min_spare_servers - the minimum number of children in 'idle'
; state (waiting to process). If the number
; of 'idle' processes is less than this
; number then some children will be created.
; pm.max_spare_servers - the maximum number of children in 'idle'
; state (waiting to process). If the number
; of 'idle' processes is greater than this
; number then some children will be killed.
; Note: This value is mandatory.
;pm = dynamic

pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes to be created when pm is set to 'dynamic'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI.
; Note: Used when pm is set to either 'static' or 'dynamic'
; Note: This value is mandatory.
pm.max_children = 128

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 20

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 15

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 30

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0

pm.max_requests = 500

; The URI to view the FPM status page. If this value is not set, no URI will be
; recognized as a status page. By default, the status page shows the following
; information:
; accepted conn - the number of request accepted by the pool;
; pool - the name of the pool;
; process manager - static or dynamic;
; idle processes - the number of idle processes;
; active processes - the number of active processes;
; total processes - the number of idle + active processes.
; The values of 'idle processes', 'active processes' and 'total processes' are
; updated each second. The value of 'accepted conn' is updated in real time.
; Example output:
; accepted conn: 12073
; pool: www
; process manager: static
; idle processes: 35
; active processes: 65
; total processes: 100
; By default the status page output is formatted as text/plain. Passing either
; 'html' or 'json' as a query string will return the corresponding output
; syntax. Example:
; http://www.foo.bar/status
; http://www.foo.bar/status?json
; http://www.foo.bar/status?html
; Note: The value must start with a leading slash (/). The value can be
; anything, but it may not be a good idea to use the .php extension or it
; may conflict with a real PHP file.
; Default Value: not set
pm.status_path = /status

; The ping URI to call the monitoring page of FPM. If this value is not set, no
; URI will be recognized as a ping page. This could be used to test from outside
; that FPM is alive and responding, or to
; - create a graph of FPM availability (rrd or such);
; - remove a server from a group if it is not responding (load balancing);
; - trigger alerts for the operating team (24/7).
; Note: The value must start with a leading slash (/). The value can be
; anything, but it may not be a good idea to use the .php extension or it
; may conflict with a real PHP file.
; Default Value: not set
;ping.path = /ping

; This directive may be used to customize the response of a ping request. The
; response is formatted as text/plain with a 200 response code.
; Default Value: pong
;ping.response = pong

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0

;request_terminate_timeout = 0

request_terminate_timeout = 240s

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0

; The log file for slow requests
; Default Value: /var/log/php-fpm.log.slow
;slowlog = /var/log/php-fpm.log.slow

; Set open file descriptor rlimit.
; Default Value: system defined value

rlimit_files = 206103

; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value

rlimit_core = 0

; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: chrooting is a great security feature and should be used whenever
; possible. However, all PHP paths will be relative to the chroot
; (error_log, sessions.save_path, ...).
; Default Value: not set
;chroot =

; Chdir to this directory at the start. This value must be an absolute path.
; Default Value: current directory or / when chroot
;chdir = /var/www

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Default Value: no

catch_workers_output = yes

; Pass environment variables like LD_LIBRARY_PATH. All are taken from
; the current environment.
; Default Value: clean env
env[HOSTNAME] = domain.ru
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /home/domain/tmp
env[TMPDIR] = /home/domain/tmp
env[TEMP] = /home/domain/tmp
;env[PHP_INI_SCAN_DIR] = /home/domain/php.ini

; Additional php.ini defines, specific to this pool of workers. These settings
; overwrite the values previously defined in the php.ini. The directives are the
; same as the PHP SAPI:
; php_value/php_flag - you can set classic ini defines which can
; be overwritten from PHP call 'ini_set'.
; php_admin_value/php_admin_flag - these directives won't be overwritten by
; PHP call 'ini_set'
; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no.

; Defining 'extension' will load the corresponding shared extension from
; extension_dir. Defining 'disable_functions' or 'disable_classes' will not
; overwrite previously defined php.ini values, but will append the new value
; instead.

; Default Value: nothing is defined by default except the values in php.ini and
; specified at startup with the -d argument
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 256M
-------------------

Обидно что в документации nginx плохо документированы некоторые важные

особенности. Я например не знал что для работы php в режиме fastcgi нужна акселерация файлов
( client_body_temp_path /var/www/virtual/arriva/tmp/tmpfs/nginx;
fastcgi_pass_request_body off;
client_body_in_file_only clean;
fastcgi_param REQUEST_BODY_FILE $request_body_file;
), и без этой самой акселерации файлы больше 200-300 килобайт не проходили обработку. После включения той самой

акселерации, о которой я узнал на сайте разработчиков php-fpm, появилась другая проблема что оказывается nginx

все временные файлы принятые пишет в доступе 0600, но т.к. у меня php-fpm pools работают от разных

пользователей, а nginx работает от пользователя apache, соответственно php-fpm сокет не мог прочитать

загруженный файл. Я изменил исходном коде nginx
(nginx-0.8.53/src/os/unix/ngx_file.c: *access = 0600;) на (*access = 0660;))
далее перекомпилировал nginx, и установил права на директорию (client_body_temp_path

/var/www/virtual/arriva/tmp/tmpfs/nginx;) равным пользователю от которого работает nginx - в данном случае

apache и группу от которой работает php-fpm pool - в данном случае arriva.
drwxr-xr-x 2 apache arriva 140 Oct 27 17:33 tmpfs
и все стало работать отлично. Думаю что данная разъяснение поможет многим кто использует подобную схему.

забыл добавить, правил еще:
nginx-0.8.53/src/os/unix/ngx_file.h
изменил строку #define NGX_FILE_OWNER_ACCESS 0600 на #define NGX_FILE_OWNER_ACCESS 0660

странно, таких проблем не испытывал

28 октября 2010 г. 13:48 пользователь copycat <nginx-forum@nginx.us> написал:
> Обидно что в документации nginx плохо
> документированы некоторые важные
>
> особенности. Я например не знал что для
> работы php в режиме fastcgi нужна
> акселерация файлов
> ( client_body_temp_path /var/www/virtual/arriva/tmp/tmpfs/nginx;
> fastcgi_pass_request_body off;
> client_body_in_file_only clean;
> fastcgi_param  REQUEST_BODY_FILE  $request_body_file;
> ), и без этой самой акселерации файлы
> больше 200-300 килобайт не проходили
> обработку. После включения той самой
>
> акселерации, о которой я узнал на сайте
> разработчиков php-fpm, появилась другая
> проблема что оказывается nginx
>
> все временные файлы принятые пишет в
> доступе 0600, но т.к. у меня php-fpm pools
> работают от разных
>
> пользователей, а nginx работает от
> пользователя apache, соответственно php-fpm
> сокет не мог прочитать
>
> загруженный файл. Я изменил исходном
> коде nginx
> (nginx-0.8.53/src/os/unix/ngx_file.c:    *access = 0600;) на (*access
> = 0660;))
> далее перекомпилировал nginx, и установил
> права на директорию (client_body_temp_path
>
> /var/www/virtual/arriva/tmp/tmpfs/nginx;) равным
> пользователю от которого работает nginx -
> в данном случае
>
> apache и группу от которой работает php-fpm pool
> - в данном случае arriva.
> drwxr-xr-x   2 apache  arriva       140 Oct 27 17:33 tmpfs
> и все стало работать отлично. Думаю что
> данная разъяснение поможет многим кто
> использует подобную схему.
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?25,144880,145271#msg-145271
>
>