Thanks for programming 101.
I'll keep your advice when my goal will be optimizing my current work,
which is not currently the case.
I do not simply want something to work here. I am fully capable of finding
workarounds whenever I need/want them.
I'll leave the 'I do not care how it works as long as it works' motto to
business-related goals ;o)
I need to understand the PHP/Nginx communication. And having searched for
it on the Web showed me a lot of unsatisfaying/dirty workarounds, no real
solution/explanation.
If anyone could enlighten me on those Nginx timeouts, I'd be more than
glad!
---
*B. R.*
On Sun, May 26, 2013 at 10:44 PM, Steve Holdoway <steve@greengecko.co.nz>wrote:
> OK, I leave you to it.
>
> However, asynchronously spawning subprocesses *will* allow you to
> parallelise the process. I'd call it design, rather than a workaround,
> but there you go (:
>
> Steve
> On Sun, 2013-05-26 at 22:38 -0400, B.R. wrote:
> > One way or another, even if an external script is called, PHP will
> > need to wait for the scripts completion, making the parallelization
> > impossible or at least useless (since, to wait for a return code of an
> > external script is still blocking).
> >
> >
> > I am not trying to find a workaround, I need to know how the
> > fastcgi_reload_timeout works (if I understood it properly), if I
> > properly disabled PHP buffering for my example case and how eventually
> > to control those timeouts.
> >
> > I'd like to address the central problem here, not closing my eyes on
> > it.
> >
> >
> > ---
> > B. R.
> >
> >
> > On Sun, May 26, 2013 at 10:24 PM, Steve Holdoway
> > <steve@greengecko.co.nz> wrote:
> > Surely, you're still serialising the transfer with a loop?
> >
> > On Sun, 2013-05-26 at 22:11 -0400, B.R. wrote:
> > > Thanks for your answer.
> > >
> > > I didn't go into specifics because my problem doesn't rely
> > at the
> > > application-level logic.
> > >
> > > What you describe is what my script does already.
> > >
> > >
> > > However in this particular case I have 16 files weighting
> > each a few
> > > MB which need to be transfered back at once.
> > >
> > >
> > > PHP allocates 30s for each loop turn (far enough to copy the
> > file +
> > > echo some output message about successes/failed completion).
> > >
> > > Nginx cuts the execution avec fastcgi_read_timeout time even
> > with my
> > > efforts to cut down any buffering on PHP side (thus forcing
> > the output
> > > to be sent to Nginx to reinitialize the timeout counter).
> > >
> > > That Nginx action is the center of my attention right now.
> > How can I
> > > get read of it in a scalable fashion (ie no
> > fastcgi_read_time =
> > > 9999999) ?
> > > ---
> > > B. R.
> > >
> > >
> > >
> > >
> > > On Sun, May 26, 2013 at 9:46 PM, Steve Holdoway
> > > <steve@greengecko.co.nz> wrote:
> > > Write a script that lists the remote files, then
> > checks for
> > > the
> > > existence of the file locally, and copy it if it
> > doesn't
> > > exist? That way
> > > no internal loop is used - use a different exit code
> > to note
> > > whether
> > > there was one copied, or there were none ready.
> > >
> > > That way you scale for a single file transfer.
> > There's nothing
> > > to be
> > > gained from looping internally - well
> > performance-wise that
> > > is.
> > >
> > > Steve
> > >
> > > On Sun, 2013-05-26 at 21:31 -0400, B.R. wrote:
> > > > No ideas?
> > > >
> > > > ---
> > > > B. R.
> > > >
> > > >
> > > > On Sat, May 25, 2013 at 1:01 PM, B.R.
> > > <reallfqq-nginx@yahoo.fr> wrote:
> > > > Hello,
> > > >
> > > >
> > > > I am trying to understand how
> > fastcgi_read_timout
> > > works in
> > > > Nginx.
> > > >
> > > >
> > > > Here is what I wanna do:
> > > >
> > > > I list files (few MB each) on a distant
> > place which
> > > I copy one
> > > > by one (loop) on the local disk through
> > PHP.
> > > >
> > > > I do not know the amount of files I need
> > to copy,
> > > thus I do
> > > > not know the total amount of time I need
> > for the
> > > script to
> > > > finish its execution. What I know is that
> > I can
> > > ensure is a
> > > > processing time limit per file.
> > > >
> > > > I would like my script not to be
> > forcefully
> > > interrupted by
> > > > either sides (PHP or Nginx) before
> > completion.
> > > >
> > > >
> > > >
> > > > What I did so far:
> > > >
> > > > - PHP has a 'max_execution_time' of 30s
> > (default?).
> > > In the
> > > > loop copying files, I use the
> > set_time_limit()
> > > procedure to
> > > > reinitialize the limit before each file
> > copy, hence
> > > each file
> > > > processing has 30s to go: way enough!
> > > >
> > > >
> > > > - The problem seems to lie on the Nginx
> > side, with
> > > the
> > > > 'fastcgi_read_timeout' configuration
> > entry.
> > > >
> > > > I can't ensure what maximum time I need,
> > and I would
> > > like not
> > > > to use way-off values such as 2 weeks or 1
> > year
> > > there. ;o)
> > > >
> > > > What I understood from the documentation
> > is that the
> > > timeout
> > > > is reinitialized after a successful read:
> > am I
> > > right?
> > > >
> > > >
> > > > The challenge is now to cut any buffering
> > occurring
> > > on the PHP
> > > > side and let Nginx manage it (since the
> > buffering
> > > will occur
> > > > after content is being read from the
> > backend). Here
> > > is what I
> > > > did:
> > > >
> > > > * PHP's zlib.output_compression is
> > deactivated by
> > > default in
> > > > PHP
> > > >
> > > > * I deactivated PHP's output_buffering
> > (default is
> > > 4096 bytes)
> > > >
> > > > * I am using the PHP flush() procedure at
> > the end of
> > > each
> > > > iteration of the copying loop, after a
> > message is
> > > written to
> > > > the output
> > > >
> > > >
> > > >
> > > > Current state:
> > > >
> > > > * The script seems to still be cut after
> > the
> > > expiration of the
> > > > 'fastcgi_read_timout' limit (confirmed by
> > the error
> > > log entry
> > > > 'upstream timed out (110: Connection timed
> > out)
> > > while reading
> > > > upstream')
> > > >
> > > > * The PHP loop is entered several times
> > since
> > > multiple files
> > > > have been copied
> > > >
> > > > * The output sent to the browser is cut
> > before any
> > > output from
> > > > the loop appears
> > > >
> > > >
> > > > It seems that there is still some unwanted
> > buffering
> > > on the
> > > > PHP side.
> > > >
> > > > I also note that the PHP's flush()
> > procedure doesn't
> > > seem to
> > > > work since the output in the browser
> > doesn't contain
> > > any
> > > > message written after eahc file copy.
> > > >
> > > >
> > > > Am I misunderstanding something about
> > Nginx here
> > > (especially
> > > > about the 'fastcgi_read_timeout'
> > directive)?
> > > >
> > > > Have you any intel/piece of advice on hte
> > matter?
> > > >
> > > > Thanks,
> > > >
> > > > ---
> > > > B. R.
> > > >
> > > >
> > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx@nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > --
> > > Steve Holdoway BSc(Hons) MNZCS
> > <steve@greengecko.co.nz>
> > > http://www.greengecko.co.nz
> > > MSN: steve@greengecko.co.nz
> > > Skype: sholdowa
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > --
> > Steve Holdoway BSc(Hons) MNZCS <steve@greengecko.co.nz>
> > http://www.greengecko.co.nz
> > MSN: steve@greengecko.co.nz
> > Skype: sholdowa
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> --
> Steve Holdoway BSc(Hons) MNZCS <steve@greengecko.co.nz>
> http://www.greengecko.co.nz
> MSN: steve@greengecko.co.nz
> Skype: sholdowa
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
