I found the solution:

My entry for
proxy_set_header Host $host:$server_port;

was originally missing the $server_port
Source : http://stackoverflow.com/questions/10168155/nginx-proxy-https-to-http-on-non-standard-port

Now it works! =D