Hi,

I have a very simple proxy config:

http {
proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m inactive=1M max_size=1000m;
proxy_temp_path /var/www/cache/tmp;
}

server {
listen 80;
server_name images.example.com;

location / {
proxy_cache s3-images-cache;
proxy_cache_key $scheme$proxy_host$uri$is_args$args;
proxy_cache_bypass $http_purge_cache;
proxy_cache_valid any 1y;
proxy_pass http://images-example.s3.amazonaws.com;
add_header X-Cache $upstream_cache_status;

proxy_intercept_errors on;
error_page 404 = @no_image;
}

location @no_image {
return 403;
}
}

1. Let's request /image.jpg.
2. Request is sent to proxy for /image.jpg (does not exists yet).
3. Backend responds with 404.
4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is called.
5. Nginx returns 403.
6. Do another request for same image and see that "X-Cache: HIT" is set. We are clearly hitting cache.

But, if we check /var/www/cache/ folder at this time we will see that there is no cache item created for this request. So does it mean Nginx keeps the cache for it in memory and forgot to write to file?

7. Let's upload /image.jpg to backend.
8. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the image instead of 403. Good.

If we check /var/www/cache/ we will see that the cache file is now finally created for this request. Also good.

9. Now here is the problem: lets request /image.jpg again.
10. Nginx returns 403 with "X-Cache: HIT". Why? So it's hitting the cache but returning something else not what is in /var/www/cache folder?? How?


My only explanation to this is it seems that Nginx is caching the response in memory and doesn't write to file when we are hitting error with our custom error_page in the proxied responses. Furthermore when using proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent requests to the same item will be using old cache which is stored in memory and not the new one created in the cache folder.

Could someone please let me know if I am doing something wrong or this really is a bug. Spent last 3 days fighting this.