Good day,

we are trying to optimise our caching strategy on cache servers that are in front of a very large dataset

200 x 1MB/s x 500 days = 1MB/s x 8’640'000'000s = 8.6 PB

We serve TV stream segments (HLS) for over 200 channels where users can fetch time shifted content. I.e. segments up to 7 days in the past (replay) or even 18 months in the past (recordings). Segments will be accessed much more rarely the older they get but the actively accessed data set is still massive. Segments never change and expire 18 months after birth.

Currently we are limited by the write rates of the SSDs in the cache servers. We can use proxy_cache_min_uses to reduce the write rate, but that also lowers our hit rates. What I would like to set up is a multi-stage cache based on different cache stores:

1) RAM disk (<1TB, very fast)
2) SSD (<10TB, fast writes)
3) HD (~100 TB)

Strategy:
a) Store every reply / requested segment in the RAM disk based cache.
b) If a segment is requested >2 times in 10min/1h store it in the SSD based cache
this may require an upstream re-fetch if it was already purged from the RAM disk cache
c) If a segment is requested >10 times in 1h store it to the HD
this may require an upstream re-fetch if it was already purged from the SSD cache

I obviously don’t want to cascade 3 instances of NGINX to pull this off..

Can anyone point me in the right direction? (Config or Code)

Thanks a lot in advance,
Joachim