Welcome! Log In Create A New Profile

Advanced

tomcat -> nginx write error while streaming lots of data

Posted by nagylzs 
tomcat -> nginx write error while streaming lots of data
April 15, 2020 05:23AM
I have an nginx proxy with this setting for a given location:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Forwarded-Host $the_host;
proxy_set_header X-Forwarded-Proto $the_scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

gzip off;

proxy_buffering off;
proxy_cache off;
proxy_pass http://my_upstream;
proxy_read_timeout 90;
proxy_redirect http://my_upstream https://www.my_fqdn.com;

A request comes in for this location. Nginx connects the upstream, which is a tomcat server. The tomcat server connects a new ZipOutputStream with the response and starts streaming files into it. Total download size can be huge. (In this concrete case it is about 2GB.)

This is what I experience:

* firefox starts downloading the zip file
* after about 500MB of download, tomcat throws this error:

```

tomcat01.0.p3kiinawp7n9@dev | org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776)
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681)
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386)
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364)
tomcat01.0.p3kiinawp7n9@dev | at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
tomcat01.0.p3kiinawp7n9@dev | at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
tomcat01.0.p3kiinawp7n9@dev | at java.base/java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:256)

```

* firefox continues to download the zip file for another minute or so, like nothing happened.
* after that, firefox throws "download failed" error and the download is aborted
* at the exact same moment, the request appears in nginx access log with HTTP 200 status code (not suprisingly, because the HTTP 200 header was set before the data streaming had started)

I think I have already disabled caching for this particular API endpoint in nginx. I'm not sure what is happening, but this is what I suspect:

* tomcat streams the data very fast
* firefox cannot keep up with this, the TCP buffer fills up
* tomcat repeatedly tries to write to the socket but it can't and SocketTimeout is raised

But I may be wrong. It is very suspicious that firefox downloads several hundred MB of data after tomcat has thrown the execption.

Questions:

* Is my nginx config correct? Should I change something in the nginx config?
* Am I missing anything else that will prevent me streaming gigabytes of data into a request like this, **possibly for hours**?
* How can I control the socket timeout on the tomcat (java) side? I could not find any method on HttpServletResponse that would change the timeout. This part of the question is not strictly related to nginx, and possibly there is nothing wrong with my tomcat setup.
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 82
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready