Nginx cache size not growing above 344GB
January 17, 2022 08:05AM
I have Nginx cache server built on Ubuntu 18 and with docker image nginx:1.19.10-alpine.

Ubuntu 18 disk usage details given below for reference
ubuntu@host_name:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 126G 0 126G 0% /dev
tmpfs 26G 1.4M 26G 1% /run
/dev/mapper/vg_system-root 193G 8.9G 176G 5% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/mapper/vg_data-srv 24T 369G 24T 2% /srv
/dev/sda1 453M 364M 62M 86% /boot
overlay 193G 8.9G 176G 5% /var/lib/docker/overlay2/64_characters_random/merged
tmpfs 26G 0 26G 0% /run/user/1646269961

Docker container details
ubuntu@host_name:~$ sudo docker ps
contnr_idxyz nginx:1.19.10-alpine "/docker-entrypoint.…" 5 days ago Up 9 hours contnr_name

Nginx Configuration for reference
user@host-name:/srv/mytool/nginx/config$ cat proxy.conf
access_log off;
root /var/log/nginx;
open_log_file_cache max=100;

log_format mytoollogformat
'$time_iso8601 $remote_addr $status $request_time $body_bytes_sent '
'$upstream_cache_status "$request" "$http_user_agent"';

proxy_http_version 1.1;
client_max_body_size 10g;

# R/W timeout for origin server
proxy_read_timeout 15m;
proxy_send_timeout 15m;

# R/W timeout for clients
client_body_timeout 15m;
client_header_timeout 15m;
send_timeout 15m;

# TODO: ssl_stapling and ssl_ciphers
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;

proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-mytool-Cache-Host $scheme://$host;

proxy_redirect off;

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mytool:10m max_size=22000g inactive=180d;
proxy_cache_key $host$uri$is_args$args$slice_range;
proxy_set_header Range $slice_range;
proxy_cache_valid 200 206 2y;
proxy_cache_revalidate on;

add_header X-Cache-Status $upstream_cache_status;

proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504 http_429;
Note: Removed server block as it comes out of context for the current issue and also due to security reason.

Let me explain my issue in detail. We have main server(proxy_passed server) which has hundreds of TBs of file which are static resources. When we set up cache server it was filling cache and serving files from cache well. But over time noticed that cache size is not increasing above 344GB

user@host_name:/srv/mytool/nginx$ sudo du -sh ./*
344G ./cache
52K ./config
1004K ./log
I have wrote a script to download about 500GB's of files. But It never increased the size of cache above 344GB.
Experiments done so far
Added max_size=100000g (along with old value min_free=1000g)
Modified max_size=22000g (set less value than /srv size which is 24TB)
removed min_free=1000g (Assuming min_free is somehow clearing cache)
"proxy_cache_valid 200 206 1h;" to "proxy_cache_valid 200 206 2y;"
For all above experiments, after configuration change i have restarted docker container and ran script of downloading 500GB of files through the cache server. But even though cache size was reaching 380 to 400 GB, but within an hour it was suddenly dropping to 344GB.

I left with no clue, why cache is not filling up completely even though i have allocated 24TB for /srv

Is it issue with Nginx? I mean for free version of Nginx there might be any limitations.. Should i go with Nginx plus. Or there might be configuration mistake.

Any guess would help me. Thanks in advance
Re: Nginx cache size not growing above 344GB
February 06, 2022 03:31AM
It is solved by modifying
keys_zone=mytool:1000m from keys_zone=mytool:10m

Please refer my answer in
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 93
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready