I'm using nginx, php-fpm, and mysql to serve a php app. Most requests
are quick, but the app has some jobs that run for a minute or more.
These are usually maintenance jobs like rebuilding a search index or
migrating large quantities of data from one format to another. When
we run a long job, the whole app blocks, and I don't understand why.

* nginx is able, during the long job, to send requests to non-php apps
just fine, so nginx is not the problem
* I was using spawn-fcgi, and had hoped that the switch to php-fpm
would solve the issue, but I've had no such luck
* at first we assumed the mysql back end was the bottleneck, but
several optimizations there have not improved the situation, for
instance, now, we have a long job that truncates one table (quickly)
then repeatedly issues insert statements on that table. Otherwise,
the only thing the script does is read other tables, so only the table
receiving the inserts might be temporarily locked, and that lock
should be released between each insert allowing other requests to read
data
* we're pretty confident the bottleneck is in the php app server
layer, which is now php-fpm
* we're currently using a unix socket for nginx <=> php-fpm
communication

Anyone have thoughts on a fix, or at least things I could try?

http://www.php.net/session_write_close ?

On 05/18/2011 08:25 AM, Zachary Crockett wrote:
> I'm using nginx, php-fpm, and mysql to serve a php app. Most requests
> are quick, but the app has some jobs that run for a minute or more.
> These are usually maintenance jobs like rebuilding a search index or
> migrating large quantities of data from one format to another. When
> we run a long job, the whole app blocks, and I don't understand why.
>
> * nginx is able, during the long job, to send requests to non-php apps
> just fine, so nginx is not the problem
> * I was using spawn-fcgi, and had hoped that the switch to php-fpm
> would solve the issue, but I've had no such luck
> * at first we assumed the mysql back end was the bottleneck, but
> several optimizations there have not improved the situation, for
> instance, now, we have a long job that truncates one table (quickly)
> then repeatedly issues insert statements on that table. Otherwise,
> the only thing the script does is read other tables, so only the table
> receiving the inserts might be temporarily locked, and that lock
> should be released between each insert allowing other requests to read
> data
> * we're pretty confident the bottleneck is in the php app server
> layer, which is now php-fpm
> * we're currently using a unix socket for nginx<=> php-fpm
> communication
>
> Anyone have thoughts on a fix, or at least things I could try?
>


--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime statistics for PHP

2011/5/18 Zachary Crockett <towynlin@gmail.com>:
> I'm using nginx, php-fpm, and mysql to serve a php app.  Most requests
> are quick, but the app has some jobs that run for a minute or more.
> These are usually maintenance jobs like rebuilding a search index or
> migrating large quantities of data from one format to another.  When
> we run a long job, the whole app blocks, and I don't understand why.
>
> * nginx is able, during the long job, to send requests to non-php apps
> just fine, so nginx is not the problem
> * I was using spawn-fcgi, and had hoped that the switch to php-fpm
> would solve the issue, but I've had no such luck
> * at first we assumed the mysql back end was the bottleneck, but
> several optimizations there have not improved the situation, for
> instance, now, we have a long job that truncates one table (quickly)
> then repeatedly issues insert statements on that table.  Otherwise,
> the only thing the script does is read other tables, so only the table
> receiving the inserts might be temporarily locked, and that lock
> should be released between each insert allowing other requests to read
> data
> * we're pretty confident the bottleneck is in the php app server
> layer, which is now php-fpm
> * we're currently using a unix socket for nginx <=> php-fpm
> communication
>
> Anyone have thoughts on a fix, or at least things I could try?
>

You should activate slowlog on FPM. For scripts which take longer than
the timeout you'll have a backtrace which can be useful for debuging
this kind of situation.

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0

; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
;slowlog = log/$pool.log.slow

Any solutions for this problem?

I have a similar problem: nginx / php-fpm / postgres.

I created a test script, that runs 20 seconds. If i call that script within a browser (chrome) and try to
recall the same script within another tab or window in chrome, the second request is blocked until the first
request returns. If i call the same script an another browser (firefox) at the same time, both requests are
handled parallel ...

The testscript is very simple:

<?php
$f = fopen('/tmp/test.txt', 'a+w');
fwrite($f, 'test');
fclose($f);

sleep(20);
?>

1. request in chrome => 'test' into file
2. request in chrome => ... nothing till the 20 seconds from the first request passed
3. request in firefox => 'test' into file even if 1. request is running

(Same behavior if 1. and 2. in firefox and 3. in chrome ...)

Thanks!

Did you ever solve this? I have just run into the ideanitcal issue. I'm just using plain php and am doing some loading of a service and so have a simple script that does:

1. fsockopen
2. fwrite (the HTTP request)
3. fread (the response)
4. fclose (sadly it doesn't seem possible to reuse the open socket for a second request/response :-( )

and then cycles around to read the resource a number of times. Precisely like you I report, in one browser, say Firefox, if this script is loading in one window in will sit and wait in another until the first window has finished. Meantime if I run a second browser, Chrome, I see the same but can run two copies of the page/script at the same time. I can also run multiple copies by saving the script a number of times, e.g. as index.php and index2.php.

So in Firefox I can run /index.php and /index2.php concurrently, but I can't run either at the same time...the second window always waits for the first. Meanwhile it's the same in Chrome and so I can for instance run 4 copies:

1. Firefox /index.php
2. Firefox /index2.php
3. Chrome /index.php
4. Chrome index2.php

but what I cannot fathom is how one window in the same browser is dependant upon the other one. It would be a neat trick to controlling multiple execution if only I could figure out what's going on!

I had issues with the same stuff and found two reasonable causes for this behaviour.

The first reason is inside PHP.
If you use Sessions in PHP and use the build-in session save_handler to write the sessions into a file, PHP will lock the file for as long as the session is opened in a process. So, the next process, trying to access the same session-file will have to wait until session_write_close() is called.
@Antony, I think this is what you wanted to say, right? :)

Here's some documentation to get a deeper understanding why:
* http://konrness.com/php5/how-to-prevent-blocking-php-requests/
* http://stackoverflow.com/questions/13651195/long-running-background-php-script-blocks-other-php-pages-until-it-is-finished

Another guy, named Morgan Cheng, tried to create a proof-of-concept for this and found out, that Firefox is blocking similar requests, if one of them has not responded yet. As far as I can see, he just compared the URL for the request, but I think Firefox compares the complete HTTP request (head and body).

Here's a link for that:
* http://stackoverflow.com/questions/2531584/how-does-session-start-lock-in-php


Solution:

So, if you can't close the session earlier, you're quite stuck here.
While reading the documentation for custom session-handlers (http://de3.php.net/manual/en/function.session-set-save-handler.php) I realized, that the read/write methods (and thus also serializing and deserializing of a session) is just done on an explicit call, not just when adding the global array $_SESSION.
So, your second call, accessing the same session as a running call, will be on hold until the other call freees the session, or it times out.
Not quite sure if php's configuration for max_execution_time counts here (you can reset this time if you want by http://php.net/manual/en/function.set-time-limit.php), but at least php-fpm shouldn't care if you have set request_terminate_timeout in your fpm-configuration - neither should nginx (see fastcgi_read_timeout).

Just to update my answer:

You are not stuck at all ;)

You can write your own session-handler, like they did in Symfony2 (https://github.com/symfony/symfony/tree/master/src/Symfony/Component/HttpFoundation/Session/Storage/Handler), and just ignore about the fact, that you can have multiple requests accessing the session.

But, if you do it like them, you may run into troubles like this guy:
http://www.matt-knight.co.uk/2011/concurrent-php-sessions/
He tried to merge the two sessions ... and found some kind of solution he was able to live with.

Here are some other good readings on that:
https://00f.net/2011/01/19/thoughts-on-php-sessions/