Hi,

We are trying to use NGINX for caching service in low bandwidth, high latency mobile networks. The service is to stream 10-sec video segments of different types ranging from 2MB to 50MB.

NGINX proxy_buffering configuration is as follows:

proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 64 4k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 64k;

The slow clients results in NGINX buffering of the response and writing data to disk as part of temporary buffering. The disk IO is causing higher TTFB and higher load time for video download.

We have tried to configure the proxy_max_temp_file_size to 0 to disable the buffering. This change results in interrupts not being balanced as shown below in the top command output - core0 and core15 is using 100%.

top - 13:53:04 up 6 days, 2:31, 5 users, load average: 6.42, 4.35, 3.84
Tasks: 370 total, 6 running, 364 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu1 : 7.9 us, 10.3 sy, 0.0 ni, 79.7 id, 0.0 wa, 0.0 hi, 2.1 si, 0.0 st
%Cpu2 : 18.7 us, 14.8 sy, 0.0 ni, 49.0 id, 0.0 wa, 0.0 hi, 17.5 si, 0.0 st
%Cpu3 : 18.7 us, 11.5 sy, 0.0 ni, 52.7 id, 0.0 wa, 0.0 hi, 17.2 si, 0.0 st
%Cpu4 : 20.1 us, 12.4 sy, 0.0 ni, 47.1 id, 0.0 wa, 0.0 hi, 20.5 si, 0.0 st
%Cpu5 : 22.2 us, 12.3 sy, 0.0 ni, 45.6 id, 0.0 wa, 0.0 hi, 19.9 si, 0.0 st
%Cpu6 : 15.1 us, 13.0 sy, 0.0 ni, 68.5 id, 0.0 wa, 0.0 hi, 3.4 si, 0.0 st
%Cpu7 : 9.6 us, 10.7 sy, 0.0 ni, 77.7 id, 0.3 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu8 : 9.9 us, 12.0 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st
%Cpu9 : 11.7 us, 11.7 sy, 0.0 ni, 73.5 id, 0.0 wa, 0.0 hi, 3.1 si, 0.0 st
%Cpu10 : 10.0 us, 11.0 sy, 0.0 ni, 74.9 id, 0.0 wa, 0.0 hi, 4.1 si, 0.0 st
%Cpu11 : 5.5 us, 12.3 sy, 0.0 ni, 80.9 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st
%Cpu12 : 1.7 us, 3.7 sy, 0.0 ni, 93.6 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu13 : 2.0 us, 3.7 sy, 0.0 ni, 93.6 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu14 : 1.4 us, 2.4 sy, 0.0 ni, 95.3 id, 0.7 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu16 : 18.0 us, 10.9 sy, 0.0 ni, 55.5 id, 0.0 wa, 0.0 hi, 15.6 si, 0.0 st
%Cpu17 : 18.8 us, 11.9 sy, 0.0 ni, 48.8 id, 0.0 wa, 0.0 hi, 20.4 si, 0.0 st
%Cpu18 : 6.4 us, 6.8 sy, 0.0 ni, 85.1 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu19 : 3.8 us, 6.5 sy, 0.0 ni, 86.3 id, 0.0 wa, 0.0 hi, 3.4 si, 0.0 st
%Cpu20 : 1.7 us, 5.1 sy, 0.0 ni, 92.2 id, 0.3 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu21 : 2.4 us, 3.0 sy, 0.0 ni, 93.9 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu22 : 1.0 us, 4.0 sy, 0.0 ni, 93.9 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu23 : 1.3 us, 3.7 sy, 0.0 ni, 94.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 11523494+total, 27725092 free, 10556008 used, 76953848 buff/cache
KiB Swap: 16777212 total, 16675408 free, 101804 used. 83445760 avail Mem

Couple of queries:
a) Why do we get unbalanced interrupts when buffering is disabled?
b) How to configure NGINX to throttle the upstream read and avoid temp buffering?

Thanks
Rajesh