Welcome! Log In Create A New Profile

Advanced

Nginx proxy cache purge process does not clean up items fast enough for new elements

October 16, 2019 05:24AM
Hi,

We have a nginx fronting our object storage which caches large objects. Objects are as large as 100GB. The nginx cache max size is set to about 3.5TB.

When there is a surge of large object requests and disk quickly fills up, nginx runs into out of disk space error. I was expecting the cache manager to purge items based on LRU and make room for the new elements, but that does not happen.

I can reproduce the problem with a simple test case:

Config:

proxy_cache_path /tmp/cache levels=1:2 keys_zone=cache_one:256m inactive=2d max_size=16G use_temp_path=off;

Test:

Run a request to download a file of 15GB, it is served correctly and stored in cache.
Run a second request to download a different file of 10GB, it will fail with something like this:

2019/10/04 11:49:08 [crit] 20206#20206: *21 pwritev() "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000001" has written only 221184 of 229376 while reading upstream, client: 127.0.0.1, server: eos_cache_filer, request: "GET...
2019/10/04 12:07:29 [crit] 21201#21201: *487 pwrite() "/tmp/cache/9/fa/a301d42ca6e5d4188c38ecf56aa3afa9.0000000002" failed (28: No space left on device) while reading upstream, client: 127.0.0.1, server: eos_cache_filer, request:

Can I tune some cache_manager parameters to make this work? Is there a way to disable buffering in such case - ideally download should not fail, it should just disable caching and buffering.

Thanks
Sachin
Subject Author Posted

Nginx proxy cache purge process does not clean up items fast enough for new elements

sachin.shetty@gmail.com October 16, 2019 05:24AM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

Maxim Dounin October 16, 2019 08:58AM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

sachin.shetty@gmail.com October 16, 2019 09:44AM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

Maxim Dounin October 16, 2019 12:50PM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

sachin.shetty@gmail.com October 17, 2019 08:13AM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

Maxim Dounin October 17, 2019 08:58AM

Re: Nginx proxy cache purge process does not clean up items fast enough for new elements

Hung Nguyen October 16, 2019 09:58AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 101
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready