I am looking for a way for allow that proxy_cache_bypass but only on a secific hostname and client IP address.

My current setup is as follows:

location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|wmv|js|css|mp3|swf|ico|flv|json|csv|txt|svg|ttf|eot|otf|cff|afm|lwfn|ffil|fon|pfm|pfb|woff|std|pro|xsf|ps|pdf|bmp)$ {
expires 1M;
add_header X-Cache-Status $upstream_cache_status;
proxy_cache_bypass $http_secret_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass http://www.domain.com;
proxy_cache static-files.domain.com;
proxy_cache_valid 200 360m;
proxy_cache_valid 302 360m;
proxy_cache_valid 404 1m;
proxy_cache_use_stale error timeout invalid_header updating;
proxy_ignore_headers X-Accel-Expires Expires;
}

How can I add to this protection so that only 1 specific IPv4 & 1 IPv6 address can bypass the poxy ?