Hi Maxim,

Thanks a lot for your reply!

I'm indeed aware of the ~8k keys per mb of memory, I was just wondering if it was handled differently when min_uses are in use, but it does indeed make sense that nginx has to keep track of it somehow, and the keys zone makes the most sense!

> Much like with any cache item, such keys are removed from the keys_zone if no matching requests are seen during the "inactive" time

That's a bummer, since that still allows memory "poisoning" - it would be awesome to have another flag for proxy_cache_path to control how long keys that have not yet reached min_uses are kept in SHM.
The benefit of this would be to say if min_uses have not been reached within let's say 5 minutes, then we purge those keys from SHM to clear up the memory.

For controlling the cache items - ideally we wanna use query strings as a part of the cache key, but still ideally prevent memory poisoning as above - the inactive flag for min_uses would be pretty useful for this - while it won't prevent it fully, we'd still be able to somewhat control memory even if people are trying to do the cache/memory poisoning.

Best Regards,
Lucas Rolff

On 17/05/2021, 16.37, "nginx on behalf of Maxim Dounin" <nginx-bounces@nginx.org on behalf of mdounin@mdounin.ru> wrote:

Hello!

On Sun, May 16, 2021 at 04:46:17PM +0000, Lucas Rolff wrote:

> Hi everyone,
>
> I have a few questions regarding proxy_cache and the use of
> proxy_cache_min_uses in nginx:
>
> Let’s assume you have an nginx server with proxy_cache enabled,
> and you’ve set proxy_cache_min_uses to 5;
>
> Q1: How does nginx internally keep track of the count for
> min_uses? Is it using SHM to do it (and counts towards the
> key_zone limit?), or something else?
>
> Q2: How long time does nginx keep this information for the
> number of accesses. Let’s say the file gets visited once in a 24
> hour period; Would nginx keep the counter at 1 for that whole
> period, or are there some set timeout where it’s “flushed”.
>
> Q3: If you have a user who decides to access files with a random
> query string on it; We want to prevent caching this to fill up
> the storage (The main reason for setting the
> proxy_cache_min_uses in the first place) – but are we gonna fill
> up the memory (and keys_zone limit) regardless; If yes – is
> there a way to prevent this?
>
> Basically the goal is to understand even just broadly how
> min_uses are counted, and possibly how to prevent memory from
> being eaten up in case someone decides to access the same URL
> once with millions of requests – if there’s any way to flush out
> the memory for example, for anything that haven’t yet reached
> the proxy_cache_min_uses if it indeed uses up memory.

The proxy_cache_min_uses basically means that if nginx sees a
request whose uses count not reached the specified limit yet, it
won't try to store the response to disk. It will, however, keep
the key in the keys_zone with the relevant information, notably
the number of uses seen so far. Quoting the proxy_cache_path
directive description (http://nginx.org/r/proxy_cache_path):

"In addition, all active keys and information about data are stored
in a shared memory zone, whose name and size are configured by the
keys_zone parameter. One megabyte zone can store about 8 thousand
keys."

Much like with any cache item, such keys are removed from the
keys_zone if no matching requests are seen during the "inactive"
time. Similarly, least recently used keys are removed if there is
not enough room in the keys_zone.

Much like with normal caching, you can control the cache key nginx
uses. If you don't want to take query string into account, you
may want to configure proxy_cache_key without the query string
(see http://nginx.org/r/proxy_cache_key).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx