the similar issue:
if use default setting,If-Modified-Since header is not fw to back end, when the cache file expired the cache server will get the file again from the backend server even if the file is not changed,it's not good for a high traffic load site.
is use proxy_set_header If-Modified-Since $http_if_modified_since,then will get unexpectedly returning 304 .
cache server cache 304 is not good i think....
Maybe the nginx should see wether the cache file exist first:
if exist then fw the if-Modified-Since header to backend,if the file has been changed, get file from the backend and overwrite the cache file,if not changes,return 304,then remain the cache file.
if no exist,then get rid of the If-Modified-Since header to get a new file from the backend