Sorry I can't offer a solution, but I seem to be experiencing a very similar issue (nginx/0.7.65).
We are using a proxy pass to a temporary host. When the host is deleted, we change it's DNS entry in order to give the client a "custom 404". Although the ttl is apparently zero, it still takes nginx 60 seconds to direct requests to the new IP address.
I am very curious if anyone knows any possible workarounds, or a likely place to look in source for hard coded timeout, etc.
Thanks,
Jay