Welcome! Log In Create A New Profile

Advanced

Re: Async operations on cached files

Piotr Sikora
December 28, 2010 11:12PM
Hi,

> I was wondering what could be a good way to spawn off processes to modify
> cached file, to perform operations that would be too long to perform as
> inline filters.
> The practical example I have is running the yui-compressor on JS and CSS
> files.
>
> I was thinking that this could be done directly on the cached files
> themselves, as a process completely decoupled from nginx

Keep in mind that cached files contain metadata, so you cannot just push
them through yui-compressor.

Other than that, using dedicated task queue seems a lot more appropriate
than doing this inside nginx.

> but this doesn't seem ideal, and cache->sh->size would probably hold an
> incorrect value for the total cache size.

Personally, I wouldn't worry too much about cache->sh->size, but if you
really do then after you're done with compressing .js/.css files, you could
purge uncompressed files from the cache using ngx_cache_purge [1] and
replace your original file with compressed one on-disk (this way nginx will
pick up compressed version from the disk on the next access).

Keep in mind that in this scenario there is (very little) chance for race
condition, so ideally you should modify ngx_cache_purge to mark content as
"being updated" instead of "deleted" before removing it from the disk and
mark it as "updated" after you've replaced it with compressed version.

> Would there be a way to build that directly into nginx?

You could fork(), process cached file in the child process and update cache
accordingly. This is far from perfect, but for a small number of tasks this
should be reasonable enough. There are some caveats while forking with "open
cache" though, check ngx_slowfs_cache [2] for details.

On the side note, you guys (CloudFlare) are talking to "remote backends" and
nginx it isn't really suited for that (at least not yet, without HTTP/1.1
and keep-alive support)... You might get a lot better results using Apache
Traffic Server, especially considering your aggressive preloading ;)

[1] http://labs.frickle.com/nginx_ngx_cache_purge/
[2] http://labs.frickle.com/nginx_ngx_slowfs_cache/

Best regards,
Piotr Sikora < piotr.sikora@frickle.com >


_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

Async operations on cached files

Matthieu Tourne 1568 December 28, 2010 08:12PM

Re: Async operations on cached files

Piotr Sikora 648 December 28, 2010 11:12PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 105
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready