The error I am getting in the logs:

"upstream prematurely closed connection while reading response header from upstream"

After the first request it works for 10-15 seconds without any problem.
According to tcpdump in the first request (the failing one) the upstream is receiving one GET and nothing more.

I found one workaround (https://hashnode.com/post/how-to-configure-nginx-to-hold-the-connections-and-retry-if-the-proxied-server-returns-502-cilngkof700iteq53ratetsry) which is adding the same upstream twice and adding the "backup"-option. The first request then fails but the second in all of my cases succeeds (checked via logs and tcpdump).

Any ideas (also some comments that could lead me to the solution would be helpful)?