Very strange, moving the directive into the server block, now blocks everything, including requests from my own server.

valid_referers server_names *.my-domain.io;
if ($invalid_referer) {
return 403;
}

Strange that when the code was copied twice into the / location bllock and php match block it worked.