I'm working on a configuration for an nginx proxy that splits requests between two upstream servers. The main reason I'm using a proxy is for SSL termination and for redundancy between the two upstream servers. Each upstream server is just a simple nginx server with identical media files stored on each.
The largest media file requested is around 2.5 megabytes. There are files larger than that, but they are requested in byte-ranges from our CDN.
I'm wondering how I should configure proxy buffering here. I noticed that the default is set to on: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering. But, I'm not sure the default values for the buffer sizes and whatnot are ideal. I've had proxy_buffering turned on for quite a while now, and have noticed that the proxy_temp directory has over 1 GB of data in it. To my understanding, this folder is used when the in-memory buffers cannot hold all of the data from the upstream, so it is written to disk.
1. If I set proxy_buffering to off, does that mean that the proxy streams the data directly to the client without buffering anything? Essentially, that would mean that an nginx worker would be "busy" on both the upstream and proxy server for the entire duration of the request, correct?
2. If I keep it on, does it make sense to change the buffer sizes so that the entire response from the upstream can fit into memory? I assume that would speed up the responses so that nothing is written to disk (slow). From my novice perspective, it seems counter intuitive to essentially read a file from upstream disk, write it to proxy disk, and then read it from proxy disk again.
What is a common use case for using proxy_buffering? Since it's a default option, I assume it's commonly used and for good reason. I'm just having a hard time applying the thought process to my specific setup.