The requests before (for more than 30sec) to the server X are ok, this is the diet request generating a 500 response (from the timeout).
Son up til this point all looks good - which is why I don't understand why nginx considers the server inactive after the first fail :)