Welcome! Log In Create A New Profile

Advanced

Strange behavior on proxy cache at high load spike

May 04, 2020 05:07PM
Hi,
this bugs me for some time now. I have nginx 1.16.0 configured as following on proxy cache:

proxy_cache_path /dev/shm/nginx_cache levels=1:2 keys_zone=proxy:1024m max_size=1024m inactive=60m;
proxy_temp_path /dev/shm/nginx_proxy_tmp;
proxy_cache_use_stale updating;
proxy_cache_lock on;
proxy_cache_lock_timeout 30s;

Most of the time all is fine and working as expected. There is some specialty in the deployment setup where some expected spikes in requests (end clients updating daily data) to few locations occur. Response size varies 1M-1.5M non-gziped. Log snippet from such spike:

[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 445984 cache: HIT request time: 50.211 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 780472 cache: HIT request time: 52.891 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 85432 cache: HIT request time: 33.284 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 57920 cache: HIT request time: 34.957 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 401096 cache: HIT request time: 49.991 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 244712 cache: HIT request time: 48.412 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 101360 cache: HIT request time: 34.955 sec
[2020-05-03T00:00:44] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 102808 cache: HIT request time: 34.753 sec
...
[2020-03-24T00:02:16] "GET /api/34/guide?date=2020-05-03 HTTP/1.0" 200 1526025 cache: HIT request time: 48.671 sec

Monitoring du on cache location shows max 1.1G, like:
1.1G /dev/shm/nginx_cache
0 /dev/shm/nginx_proxy_tmp

After 2minutes response 'stabilizes' with correct size (in this example 1526025). Problem is also amplified due clients validate response and retry progressively if corrupted.

There are no weird log lines in error log or linux (centos) messages, also there is no cache 'updating', just hits (I guess this omits upstream servers issue). Is it possible we have issue with reading cached entries from /dev/shm during peak times?

I would kindly ask for hints where possibly to start looking and debugging?
Big thanks in advance
Subject Author Posted

Strange behavior on proxy cache at high load spike

abr3 May 04, 2020 05:07PM

Re: Strange behavior on proxy cache at high load spike

J.R. May 04, 2020 08:56PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 88
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready