Hello!

On Mon, May 22, 2017 at 07:02:04PM +0000, Karstens, Nate wrote:

> Maxim,
>
> I'd be happy to explain. Our application is actually relying
> more on the change to support to SO_SNDBUF option, but I noticed
> the SO_RCVBUF option there and thought it was worth exposing
> that at the same time.
>
> The application we're having issues with is essentially a file
> transfer through a proxy to a relatively slow storage medium.
> Because we're using a proxy there are three buffers involved on
> the receive end of the HTTP request: 1) receive buffer on
> external nginx socket, 2) send buffer in nginx proxy module, and
> 3) receive buffer on proxied server. So, in a system where each
> buffer is a maximum of 5 MB, you can have 3 x 5 = 15 MB of data
> in the TCP buffers at a given point in time.

Send buffer on the client contributes to overral buffering as
well, and probably should be counted too. But, frankly, 5MB is a
lot, and much higher than typical default, and may result in
problems by itself. See
https://blog.cloudflare.com/the-curious-case-of-slow-downloads/
for a description of a problem Cloudflare faced with such socket
buffer sizes.

> In most circumstances, I don't think this would be a problem.
> However, writing the data is so slow that the HTTP client times
> out waiting for a reply (which is only sent once all data has
> been written out). Unfortunately, we cannot solve this by
> increasing the client's timeout. We found that reducing the size

Are you using some custom client, or a browser?

Even with buffers as huge as 3x5MB, a really slow backend will be
required to trigger a timeout in a typical browser, as browsers
seems to happily wait for 60+ seconds (actually, much more, but 60
seconds is a default timeout in nginx). This means that backend
needs to be slower than 250 KB/seconds for this to become a
problem even without any socket tuning, and this sounds more like
an 1x CD than even a class 2 SD-card.

> of each buffer -- using the "rcvbuf" parameter to the "listen"
> directive lets us configure SO_RCVBUF for the first and third
> sockets mentioned above, and this patch lets us configure
> SO_SNDBUF of the second socket -- reduces the time between when
> the client sends the last byte of its request and when it
> receives a reply, ultimately preventing the timeout. We would
> prefer not to adjust the system's default sizes for these
> buffers because that negatively impacts performance on other
> applications used by the system.
>
> Although this seems like a fairly-specific use-case, I think it
> can be generalized as: 1) the client cannot wait indefinitely
> after sending the last byte of the request, 2) the server must
> process all data before it can generate a reply, and 3) the
> server processes data relatively slowly. This seemed general
> enough that it was worth adding the functionality for our own
> use and thought it might be applicable to other users as well.

Normally, nginx reads the whole request from the client, and only
after that starts sending it to a backend server. This
effectively means infinite buffering, and certainly will trigger a
timeout if the backend server is not able to process the request
in a reasonable time. Socket buffer sizes may become important
when using "proxy_request_buffering off" and/or non-http
connections (e.g., WebSocket ones), but these are specific by
itself.

Overall, thank you for the patch, but it looks like something very
specific for your particular use case. We would like to avoid
introducing this into nginx, at least till there are more requests
for this.

I would also recommend to take a closer look at your setup.
Numbers you've provided suggest that there may be something wrong
elsewhere, and the fact that smaller buffers fix the problem may
be unrelated.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel