Hi,

To realize a distributed caching layer based of disk-speed and storage, I have prepared the following configuration with an SSD-cache and HDD-cache.

http {

add_header X-UpStream-Server-Cache-Status $upstream_cache_status;

# proxy caching configurations
proxy_cache_path /tmp/myssd1 keys_zone=myssd1:1m levels=1:2 inactive=60m;

proxy_cache_path /tmp/myhdd1 keys_zone=myhdd1:10m levels=1:2 inactive=60m;

proxy_cache_key "$request_uri";

upstream mylocalhdd {
server 127.0.0.1:82;
}

upstream myorigin {
server 192.168.1.35:80;
}

# Server-block1 - will cache in SSD if we get a minimum of 5 requests.
server {
listen 80;
server_name example.com www.example.com;

# Look up in ssd cache. If not found, goto hdd cache.
location / {
proxy_pass http://mylocalhdd;
proxy_cache myssd1;
proxy_cache_min_uses 5;
}
}

# Server-block2 (acting as local upstream) - will fetch from origin and cache in HDD
server {
listen 82;
server_name example.com www.example.com;

# Look up in hdd cache. If not found, goto origin
location / {
proxy_pass http://myorigin;
proxy_cache myhdd1;
}
}

}

The smaller, yet faster SSD-cache will store content only if I get at least 5 requests for the URL. On the other hand, the larger HDD cache will every request.

The flow is straight-forward:

i) Client's first request is handled by server at port 80
ii) It looks in SSD cache. Upon a cache-miss, it proxies to local upstream at port 82.
iii) The local server at port 82, looks in HDD cache. upon cache miss, it fetches from origin, and adds to HDD-cache and sends the response back to first server.
iv) Server1 does not add content yet to SSD-cache (as min_uses is not reached), and sends response to client.
v) For the next 3 requests, the server1 would see an SSD-cache-miss, but server2 produces an HDD-cache-hit.
vi) For the 5th request, the server1, will also add the response in SSD-cache, as min_uses criteria is met.
vii) For the 6th request onwards, the server1 itself will serve the request from SSD-cache itself. No request is sent to local upstream.

I have added $upstream_cache_status in the response.

For the first request, I see the header twice in the response: This seem to correspond to MISS in both front-end SSD-cache and back-end HDD-cache.

< X-UpStream-Server-Cache-Status: MISS
< X-UpStream-Server-Cache-Status: MISS

For the next 4 requests, I see the following: This seems to correspond to MISS in the from-end SSD-cache and HIT in the back-end HDD-cache.

< X-UpStream-Server-Cache-Status: HIT
< X-UpStream-Server-Cache-Status: MISS

For the 6th request, I see the following:

< X-UpStream-Server-Cache-Status: HIT
< X-UpStream-Server-Cache-Status: HIT

Why am I getting the header twice for the 6th request. In this case, the request is HIT by the SSD cache itself, and there is no request sent to local upstream also.

So, shouldn't I be getting only one instance of the header in the response?

Thanks
Rajesh