Hello,

I hope everyone is doing well and staying healthy!!

I recently found this forum while investigating a performance issue I'm having when proxy buffering is enabled. I've been looking and trying different solutions but none of them are working for me, so I hope your wisdom can help me to understand and figure out what the issue is.

Here is a description of my environment and the problem that I'm having. I have two servers working as a reverse proxy, one server only responds to requests on port 80, while the other one serves SSL certificates on port 443.

Server A:
Listens to port 80
nginx version: openresty/1.13.6.2
Rhel 6.10
Physical Memory: 64GB
CPUs: 55 cores @ 2.00GHz

Server B:
Listens to port 443
nginx version: openresty/1.13.6.2
Rhel 6.10
Physical Memory: 128GB
CPUs: 55 cores @ 2.00GHz

I enabled caching and noticed some issues with the buffers, so I tunned up the parameters in the SSL server and started with this configuration:

proxy_buffering on;
proxy_temp_path /proxy_buffer/tmp 1 2;
proxy_buffer_size 64k;
proxy_buffers 64 4k;
proxy_busy_buffers_size 84k;

With this setup, the server is running a little bit better, CPU is at 0%, memory at 4% and load around 9%. I noticed load is high because there are still some requests that don't fit into the buffers and are being written to disk (I/O wait is around 0.8%). Actually, after one week with this configuration running the proxy_temp_path directory has grown up to 30gb of disk usage. If I disable buffering the load goes down to 1%.

I also checked Nginx stats and this SSL server has around 2500 active connections per minute and it is getting around 15000 requests per minute.

The history on the server A is totally different, it is busier than the SSL one, it currently has buffering disabled and it's getting around 29K requests per minute with 6500 active connections. CPU is at 2%, Memory is at 5% and load average is 0.8% with no I/O wait.

If I enable buffering on this host with the same aforementioned configuration the server begins to have performance problems; in less than two hours memory utilization gets to 100% and load average spikes up to 50 ~ 60%.

I've been trying different setups like fewer buffers of bigger size or more buffers with a bigger size and I am not able to find a proper solution to make this work.

I'd really appreciate any advice or recommendation you can guys can give me, I'd like to know what would be the buffer size recommended based on the stats and hardware specs I mentioned above.

Thanks,