I'm having problem with I/O performance. I'm running nginx as caching reverse proxy server.
When cache size on disk exceeds max_size cache manager starts working, but it causes two problems occur:

1) I/O %util reach 100% and nginx starts dropping connections
2) cache manager process dosen't unlink files speed enough to delete old file. So cache becomes bigger util the space on disk ends.

Can you give me an idea how can I solve those problems. Below are some details.

#build on 20x 300GB SAS disks with 2 SSDs for Cachecade.

# storcli64 /c0 show
VD LIST :
=======

----------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
----------------------------------------------------------------
1/2 RAID60 Optl RW Yes RaWBC R ON 4.357 TB
2/1 Cac0 Optl RW Yes RaWTD - ON 557.875 GB
----------------------------------------------------------------

# mount
/dev/sdb1 on /cache type ext4 (rw,noatime,data=ordered)

# df -h /dev/sdb1
/dev/sdb1 4.3T 3.2T 828G 80% /cache


# for pid in `pgrep nginx `;do ionice -p $pid ;done
unknown: prio 4 <- master
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0 <- workers
idle <- cache manager

# grep proxy_cache_path nginx.conf
proxy_cache_path /cache zone=my-cache:20000msize=3355443m

# netstat -sp|grep -i drop
6335115 SYNs to LISTEN sockets dropped

# iostat -dx 1 /dev/sdb |grep ^sdb | awk '{print $14}'
24.40
31.20
26.80
23.60
26.80
16.00
34.80
35.20
29.60
...
14.40
15.60
11.60
16.00
17.20
18.00
17.20
42.00
90.80 <- cache manager process starts
100.00
100.00
29.20
100.00
100.00
100.00
52.00
100.00
100.00
100.00