Welcome! Log In Create A New Profile

Advanced

Re: [PATCH 00 of 15] Serve all requests from single tempfile

Roman Arutyunyan
February 08, 2022 06:20AM
Hi,

On Mon, Feb 07, 2022 at 01:27:15PM +0100, Jiří Setnička via nginx-devel wrote:
> Hello,
>
> > Thanks for sharing your work. Indeed, nginx currently lacks a good solution
> > for serving a file that's being downloaded from upstream. We tried to address
> > this issue a few years ago. Our solution was similar to yours, but instead
> > of sharing the temp file between workers, we moved the temp file to its
> > destination right after writing the header. A new bit was added to the header
> > signalling that this file is being updated.
> >
> > The biggest issue with this kind of solutions is how we wait for updates in
> > a file. We believe that polling a file with a given time interval is not a
> > perfect approach, even though nginx does that for cache locks.
>
> polling is done only on the ngx_http_file_cache_tf_node_t struct in the
> shared memory (see patch 09 of 15, where c->length is updated from
> c->tf_node->length and then this length is compared with
> c->body_sent_bytes), not on the file itself. Length in the tf_node is
> updated with each write from the primary request (see patch 05 of 15).
>
> It is better than polling individual files, but I agree it is still polling,
> which isn't great.
>
> > [...]
> > Another approach would be to create an
> > inter-worker messaging system for signalling file updates.
>
> We were thinking about creating something like that but we buried this idea
> because it seems quite complex to do it right and reliable. And polling the
> tf_node in the shared memory (with very low proxy_cache_tempfile_loop) works
> sufficiently good.
>
>
> > It's good to know the solution works for you. Please keep us posted about
> > future improvements especially the ones which would avoid polling and decrease
> > complexity.
>
> We would be happy to get this patch to the mainline nginx in the future, so
> that all nginx users could benefit from it.
>
> We will be thinking about avoiding polling and implementing some
> inter-worker messaging, but it may be some time, because it seems quite
> complex. Could you share some hints about how do you thing it would be best
> to implement it in the worker's event loop?

Even though I have some ideas, they need to be checked first. I can only say
this should be a core nginx feature.

> Jiří Setnička
> CDN77
>
> _______________________________________________
> nginx-devel mailing list -- nginx-devel@nginx.org
> To unsubscribe send an email to nginx-devel-leave@nginx.org

--
Roman Arutyunyan
_______________________________________________
nginx-devel mailing list -- nginx-devel@nginx.org
To unsubscribe send an email to nginx-devel-leave@nginx.org
Subject Author Views Posted

[PATCH 00 of 15] Serve all requests from single tempfile

Jiří Setnička via nginx-devel 258 January 28, 2022 11:36AM

[PATCH 01 of 15] ngx core - obtain number appended to the temp file name

Jiří Setnička via nginx-devel 52 January 28, 2022 11:38AM

[PATCH 02 of 15] ngx core - ensure that tempfile number never be 0

Jiří Setnička via nginx-devel 60 January 28, 2022 11:40AM

[PATCH 03 of 15] Cache: Shared memory for tempfile nodes

Jiří Setnička via nginx-devel 84 January 28, 2022 11:42AM

[PATCH 04 of 15] Cache: tf_node for tracking opened tempfile

Jiří Setnička via nginx-devel 49 January 28, 2022 11:44AM

[PATCH 05 of 15] http upstream & file_cache: store temp file number and length in tf_node

Jiří Setnička via nginx-devel 54 January 28, 2022 11:46AM

[PATCH 07 of 15] Tempfiles: Wait handlers

Jiří Setnička via nginx-devel 51 January 28, 2022 11:48AM

[PATCH 06 of 15] Configuration for tempfiles serving

Jiří Setnička via nginx-devel 88 January 28, 2022 11:50AM

[PATCH 09 of 15] Tempfiles: Sending data from tempfile

Jiří Setnička via nginx-devel 49 January 28, 2022 11:52AM

[PATCH 08 of 15] Tempfiles: Mechanism of opening tempfiles used for serving paralell requests

Jiří Setnička via nginx-devel 51 January 28, 2022 11:54AM

[PATCH 10 of 15] Tempfiles: Setup event handlers in ngx_http_upstream.c

Jiří Setnička via nginx-devel 50 January 28, 2022 11:56AM

[PATCH 11 of 15] Tempfiles: reset c->body_start when updating a tempfile

Jiří Setnička via nginx-devel 53 January 28, 2022 11:58AM

[PATCH 12 of 15] Tempfiles: Expired tempfiles

Jiří Setnička via nginx-devel 78 January 28, 2022 12:00PM

[PATCH 13 of 15] Tempfiles: Skip cached file if there is already newer tempfile

Jiří Setnička via nginx-devel 106 January 28, 2022 12:02PM

Re: [PATCH 13 of 15] Tempfiles: Skip cached file if there is already newer tempfile

Vadim Fedorenko 67 January 30, 2022 07:38PM

Re: [PATCH 13 of 15] Tempfiles: Skip cached file if there is already newer tempfile

Jiří Setnička via nginx-devel 50 February 07, 2022 06:22AM

[PATCH 14 of 15] Tempfiles: Set send_timeout inside ngx_http_cache_send

Jiří Setnička via nginx-devel 50 January 28, 2022 12:04PM

[PATCH 15 of 15] Use cache status UPDATING when serving from tempfile

Jiří Setnička via nginx-devel 53 January 28, 2022 12:06PM

Re: [PATCH 00 of 15] Serve all requests from single tempfile

Jiří Setnička via nginx-devel 44 February 07, 2022 06:18AM

Re: [PATCH 00 of 15] Serve all requests from single tempfile

Roman Arutyunyan 52 February 07, 2022 06:32AM

Re: [PATCH 00 of 15] Serve all requests from single tempfile

Jiří Setnička via nginx-devel 49 February 07, 2022 07:30AM

Re: [PATCH 00 of 15] Serve all requests from single tempfile

Roman Arutyunyan 72 February 08, 2022 06:20AM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 54
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready