I have more info on the system state at the time the CLOSE_WAIT connections went sky rocketing

Memory
#######################################

KiB Mem : 13174569+total, 8684164 free, 28138264 used, 94923264 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 86984112 avail Mem
########################################

CPU
#############################################
top - 16:52:02 up 4 days, 5:37, 1 user, load average: 3.68, 5.14, 6.06
Tasks: 935 total, 3 running, 932 sleeping, 0 stopped, 0 zombie
%Cpu(s): 22.9 us, 8.0 sy, 0.0 ni, 66.2 id, 2.3 wa, 0.0 hi, 0.6 si, 0.0 st
##############################################

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3825721 nobody 20 0 3106072 2.4g 3960 R 105.6 1.9 0:57.32 nginx: worker process


Nginx worker process was using 105% CPU


!---------------------------------- netstat by state
16234 CLOSE_WAIT
6 CLOSING
1195 ESTABLISHED
5 FIN_WAIT1
22 FIN_WAIT2
14 LAST_ACK
63 LISTEN
447 SYN_RECV
10 SYN_SENT
1875 TIME_WAIT


!---------------------------------- netstat by state
20139 CLOSE_WAIT
2 CLOSING
23482 ESTABLISHED
17703 FIN_WAIT1
4 FIN_WAIT2
19 LAST_ACK
63 LISTEN
863 SYN_RECV
14 SYN_SENT
130 TIME_WAIT


Nginx Process
###################################
root 1724190 1.9 1.9 2835060 2555304 ? Ss 10:17 7:48 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody 3825721 10.6 1.9 3106072 2561764 ? Rl 16:42 1:02 \_ nginx: worker process
nobody 3825723 0.0 1.9 2835220 2549460 ? S 16:42 0:00 \_ nginx: cache manager process
####################################



As soon as Nginx was stopped and apache was made the frontend web server the connections changed as below

############
!---------------------------------- netstat by state
1 CLOSE_WAIT
2 CLOSING
389 ESTABLISHED
35111 FIN_WAIT1
2 FIN_WAIT2
44 LAST_ACK
42 LISTEN
1 SYN_RECV
12 SYN_SENT
86 TIME_WAIT
##################################

FIN_WAIT1 -- was high , but ESTABLISHED and CLOSE_WAIT went normal

The overall CPU usage and memory remained fine and CPU utilization was normal too


A difference that I can tell with Nginx and httpd on the server is that Nginx is configured for http/2 while httpd doesn't have http2 module enabled

The following is how the listen is configured for the default server in Nginx

server {
listen x.x.x.x:80 default_server backlog=16384 reuseport deferred;

server {
listen x.x.x.x:443 default_server ssl backlog=16384 reuseport deferred;


the individual vhosts have http2 also enabled