Okay, I did a little more digging into the issue and I think I've narrowed down the problem a little more.
FYI I'm running nginx 1.23.3
So my location matchs are fine and working as expected. I even have an "if" statement within said location and it works as expected...
if ($invalid_referer) { return 403; }
But then I started to experiment with add_header within the location block, adding a custom tag just to see what was going on and where responses were coming from.
So in another location for image serving, when nginx serves the image directly via try_files, then the temporary header is included. However if it gets passed to the @proxy then the header (within the location block) is missing. However, higher level add_header tags are present.
This seems to fall in line with my proxy cache settings within the location block not being processed, but instead using the higher level ones.
Is this by design, or a bug, or am I missing something?