Welcome! Log In Create A New Profile

Advanced

One last try - large long-running worker tasks

November 09, 2020 08:56PM
Hi all, I've asked this before with no response, trying one last time
before I just make something work.

I'm making a custom module for nginx that does a number of things but
one of the actions is a long-running (in the nginx sense) task that
could produce a large response. I've already got proper processing
around using worker tasks for the other long-running operations that
have small datasets, but worry about accumulating a large amount of
memory in a buffer chain for the response. Ideally it would drain as
fast as the client can consume it and throttle appropriately, there
could conceivably be gigabytes of content.

My choices (besides blowing all of the memory in the system) are:

- write to a temporary file and attach a file buffer as the response,
less than ideal as it's essentially translating a file to begin with,
so it's a lot of disk I/O and performance will be less than stellar.
From what I can tell, this is one of the models for the various CGI
systems for caching, although in my case caching is not of use

- somehow hook into the eventing system of nginx to detect the write
transitions and implement flow control directly using threading
conditionals. I've tried this for a few weeks but can't figure out
the 'right' thing to make the hooks work in a separate module without
changing the core nginx code, which I'm loathe to do (unless you are
looking for someone to contribute such a solution, but I'd probably
need some initial guidance)

- attach a kernel pipe object (yah yah, won't work on Windows, don't
care) to each of my worker instances and somehow 'connect' that as an
upstream-like resource, so that the nginx event loop handles the
read/write consumption and the thread automatically blocks when full
on the kernel pipe. Would need some jiggery to handle reuse and
start/end markers. Also not clear if I can override the connection
model for the upstream without again changing core nginx server code

Any thoughts? Not looking for code here (although telling me to look
at the blah-blah-blah that does exactly this would be awesome), but if
someone who is more familiar with the inner workings of the nginx data
flow could just say which solution is a non-starter (so I don't waste
time trying to make it work) or even which would be a suitable
solution would be awesome!

jmh
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

One last try - large long-running worker tasks

jmheisz 71 November 09, 2020 08:56PM

Re: One last try - large long-running worker tasks

itpp2012 13 November 10, 2020 03:18AM

Re: One last try - large long-running worker tasks

Dipl. Ing. Sergey Brester 9 November 10, 2020 07:50AM

Re: One last try - large long-running worker tasks

jmheisz 16 November 10, 2020 08:36AM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 68
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready