OK finally figured it out I think.

Seems it may be to do with the scope of the user set variables (not sure) but after moving adding proxy_no_cache and proxy_cache_bypass to the same location instead of the centrally called ones, It seems to be working as expected.

The hint was in agentzh's original post

location / {
set_by_lua $no_cache '
if ngx.var.cookie_COOKIE == "" or ngx.var.cookie_COOKIE == nil then
return 1
end
return ""
';
[b]proxy_cache_bypass $no_cache;[/b]
proxy_pass ...
}