Oh. Ok, good to know about the default temp file and buffers.
Just checked and I think the 'large' file we are downloading is 800mb.
We don't have proxy_cache or proxy_store set. We do have
proxy_temp_file_write_size 250m;

We ended up doing a test where 9 of those large files were all on server1,
and it continued to round robin requests.

Is that temp_file_size essentially per connection? If so, then if the file
is only 800mb, then perhaps that makes sense and we are indeed
closing the upstream and it is just buffered waiting to finish sending to
client.

I'll try:
1) using a file larger than 1gb (just to see if we can force it to be
larger than the possible buffer).
2) Still do the netstat.. I think that will tell us a whole lot.
3) simulate with limit_rate

Thanks again! I think we might be on to something.




On Wed, Dec 23, 2020 at 5:15 PM Maxim Dounin <mdounin@mdounin.ru> wrote:

> Hello!
>
> On Wed, Dec 23, 2020 at 04:42:49PM -0500, Kenneth Brooks wrote:
>
> > We did think that perhaps it was buffering.
> > However, in our case, the "large" request is gigs in size, so there is no
> > way that it is buffering that whole thing. I think our buffers are pretty
> > small.
> > Unless there is some absolute black magic that will buffer what it can,
> > close the upstream, then open it again to ask for more of that file. :)
>
> By default nginx can buffer up to slightly more than 1 gigabyte
> (proxy_max_temp_file_size + various in-memory buffers). Further,
> with proxy_cache (or proxy_store) the proxy_max_temp_file_size limit
> is ignored, so nginx can buffer arbitrary responses.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx