Hello Valentin,

> 1. Write socket buffer in kernel on node.js side where node.js writes data.

we can throw this out from equation, as I measure my end time by the event when socket is closed on nodejs side, (I use http1.0 from nginx to node to make it simple for this case).

> 2. Read socket buffer in kernel for node.js connection from what nginx reads data.

SO_RCVBUF shouldn't be over 64KB by default. What does nginx use, is there a config that controls it?.. still this shouldn't be a big issue, I'm fine if there is such a constant buf.


> 3. Heap memory buffer to that nginx reads data from kernel socket buffer (controlled by proxy_buffers
> and proxy_buffer_size directives).
>
> No buffering here means that nginx doesn't keep that data in buffers
> for some time, but writes it immediately to write socket buffer in kernel
> for client connection.

I'm trying to configure these to be skipped or used to minimum. E.g. I don't wan any data to be held in these.

> 4. Write socket buffer in kernel for client connection where nginx writes data.

SO_SNDBUF shouldn't be over 64KB by default, perhaps nginx changes it as well. What's the value that nginx uses and is there a config that controls it?

> 5. Read socket buffer in kernel for client connection from what wget reads data.

We can throw this out from equation, we may assume these aren't used, as for my test I use final time when wget finishes and prints stats. There is obviously highly unlikely chance that wget actually reads data twice faster from network, but shows slower speed in it's cli results and "waits" for data even though it's already received and is in local buffers. This is totally dumb and I don't think this might happen, but I could check with wireshark just in case.


In short, these could affect my case: SO_RCVBUF, SO_SNDBUF on nginx side and whatever buffering nginx uses for handling data. I run that same test with 25MB data and I got totally identical result: 12.5MB was buffered on nginx side. That stuff that could affect my case cannot really add up to 12.5MB and 10 minute of time.
There is a wild possibility that tcp window scaling resulted in some huge window on node->nginx side and ended up storing that 12MB in tcp window itself but i'm not sure if TCP window should be accounted into these SO_RCVBUF or that RCVBUF is extra data on top of internals of TCP.

So,.. any ideas how come nginx ends up buffering 12.5MB data?