So upon further investigation, I noticed that changing it the zone name to "v$binary_remote_addr" actually breaks nginx and prevents configuration being reloaded at all even though the reload is going through successfully until the next restart, and after that, it begins dropping the connections again on reload.