I am running a small image-hoster with PHP-FPM 5.3.4 + Lighttpd 1.5/
svn on Debian Lenny.

The "script-flow" is the following:

-> 25MB ZIP gets uploaded by user (via simple html-form / as HTTP-
POST)
-> ZIP containts 19 Jpeg-Images - each approx. 1.4 MB
-> via Imagick (PHP-Extension - not CLI) thumbs are created
-> via ssh2 (PHP-Extension - cURL 7.21.3 and libssh2 1.2.7 self
compiled) the images are uploaded to external hosts (CDN)

In general this works fine with smaller ZIPs, and ZIPs with 25MB but
containing smaller files (< 1MB) - so I am guessing the script as such
is ok.

But with the above example, I get as final page a blank page and the
response header reads "HTTP/0.9 200...". Normally if you get a PHP-
error (error_reporting / display_errors all on), the response header
will be "HTTP/1.1 500...".

All thumbs are generated fine and are uploaded to their final
destination.

-> I checked max_execution_time / max_input_time - there should be no
timeouts, as the complete process is finished in max 10min - limits
are set to 30min
-> I checked the memory-usage via xhprof - the usage is fairly low (<
5MB) - memory-limit is set to 128M
-> I checked all kind of error-logs - did not find anything

I have got this error, ever since I switched from Apache/mod_php to
Lighttpd/php-fpm - I first thought it had to do with the limits/
php.ini - but I migrated everything and even loosened it up.

any ideas?

php-fpm is standalone SAPI, could not be used with mod_php

On Sun, Dec 26, 2010 at 8:55 PM, Günter G. <guenter@grodotzki.ph> wrote:

> I am running a small image-hoster with PHP-FPM 5.3.4 + Lighttpd 1.5/
> svn on Debian Lenny.
>
> The "script-flow" is the following:
>
> -> 25MB ZIP gets uploaded by user (via simple html-form / as HTTP-
> POST)
> -> ZIP containts 19 Jpeg-Images - each approx. 1.4 MB
> -> via Imagick (PHP-Extension - not CLI) thumbs are created
> -> via ssh2 (PHP-Extension - cURL 7.21.3 and libssh2 1.2.7 self
> compiled) the images are uploaded to external hosts (CDN)
>
> In general this works fine with smaller ZIPs, and ZIPs with 25MB but
> containing smaller files (< 1MB) - so I am guessing the script as such
> is ok.
>
> But with the above example, I get as final page a blank page and the
> response header reads "HTTP/0.9 200...". Normally if you get a PHP-
> error (error_reporting / display_errors all on), the response header
> will be "HTTP/1.1 500...".
>
> All thumbs are generated fine and are uploaded to their final
> destination.
>
> -> I checked max_execution_time / max_input_time - there should be no
> timeouts, as the complete process is finished in max 10min - limits
> are set to 30min
> -> I checked the memory-usage via xhprof - the usage is fairly low (<
> 5MB) - memory-limit is set to 128M
> -> I checked all kind of error-logs - did not find anything
>
> I have got this error, ever since I switched from Apache/mod_php to
> Lighttpd/php-fpm - I first thought it had to do with the limits/
> php.ini - but I migrated everything and even loosened it up.
>
> any ideas?

Imagick leaks memory like a sieve. I would look there for your
problems, everything else sounds like you're on top of things.

Is it possible for you to shift the image processing to an
asynchronous cron job? This way the user always gets a valid response
as long as the upload succeeds, and the image processing does not tie
up your fpm workers. The last time I found myself having to do
synchronous imagick processing, I put those workers in their own pool
that recycled after one job - this let everything else run normally
and helped me isolate the misbehaving thumbnail generation script.

- ammon

On Dec 26, 2010, at 7:54 AM, Günter G. <guenter@grodotzki.ph> wrote:

> I am running a small image-hoster with PHP-FPM 5.3.4 + Lighttpd 1.5/
> svn on Debian Lenny.
>
> The "script-flow" is the following:
>
> -> 25MB ZIP gets uploaded by user (via simple html-form / as HTTP-
> POST)
> -> ZIP containts 19 Jpeg-Images - each approx. 1.4 MB
> -> via Imagick (PHP-Extension - not CLI) thumbs are created
> -> via ssh2 (PHP-Extension - cURL 7.21.3 and libssh2 1.2.7 self
> compiled) the images are uploaded to external hosts (CDN)
>
> In general this works fine with smaller ZIPs, and ZIPs with 25MB but
> containing smaller files (< 1MB) - so I am guessing the script as such
> is ok.
>
> But with the above example, I get as final page a blank page and the
> response header reads "HTTP/0.9 200...". Normally if you get a PHP-
> error (error_reporting / display_errors all on), the response header
> will be "HTTP/1.1 500...".
>
> All thumbs are generated fine and are uploaded to their final
> destination.
>
> -> I checked max_execution_time / max_input_time - there should be no
> timeouts, as the complete process is finished in max 10min - limits
> are set to 30min
> -> I checked the memory-usage via xhprof - the usage is fairly low (<
> 5MB) - memory-limit is set to 128M
> -> I checked all kind of error-logs - did not find anything
>
> I have got this error, ever since I switched from Apache/mod_php to
> Lighttpd/php-fpm - I first thought it had to do with the limits/
> php.ini - but I migrated everything and even loosened it up.
>
> any ideas?

On Sun, Dec 26, 2010 at 10:36 PM, Ammon Lauritzen <allaryin@gmail.com> wrote:
> Imagick leaks memory like a sieve. I would look there for your
> problems, everything else sounds like you're on top of things.
>

This is the first time I heard this. Care providing reference link?

On Dec 26, 4:36 pm, Ammon Lauritzen <allar...@gmail.com> wrote:
> Imagick leaks memory like a sieve. I would look there for your
> problems, everything else sounds like you're on top of things.

but wouldn't I get a PHP-error when my script run out of assigned
memory?

like said above, I checked my scripts with xhprof and the memory
consumption was very low.

> Is it possible for you to shift the image processing to an
> asynchronous cron job? This way the user always gets a valid response
> as long as the upload succeeds, and the image processing does not tie
> up your fpm workers. The last time I found myself having to do
> synchronous imagick processing, I put those workers in their own pool
> that recycled after one job - this let everything else run normally
> and helped me isolate the misbehaving thumbnail generation script.

yes that would be a solution but would be not very user friendly as it
would slower things down a lot. I still believe with enough hardware
(which I think we have, max 50% RAM usage, with 8 cpus only a load of
3-5) this should be manageable.

Is this maybe a timeout problem with the fcgi process? ->
http://wiki.nginx.org/HttpFcgiModule#fastcgi_read_timeout - though
unfortunately lighttpd does not have this config option.

you can in fact have multiple SAPIs running at the same time (e.g.
apache + mod_php / cli / fcgi / etc.) - though this is not the case
here anyway ;)

e.g. on the server (8GB RAM / Intel(R) Xeon(R) CPU E5430 @ 2.66GHz)
only lighttpd with php-fpm is running.


On Dec 26, 3:53 pm, 陈智宏 <zhihong.chen...@gmail..com> wrote:
> php-fpm is standalone SAPI, could not be used with mod_php
>

ok I figured out this is clearly a lighttpd + php-fpm problem which
you reproduce with a simple one-liner, I filed bug report here:
http://redmine.lighttpd.net/issues/2282 and hope this will get soon
fixed.