Thanks Maxim

We enabled the upstream_request_time on both the server which shows response time less than a sec for Upstream request.
It doesn't seems to be issue with Upstream Server .
Even for the request which are HIT response time on the server on which "Writing" is more varies from 10 sec to 60 sec and even more while on other its less than 2 sec either its MISS or HIT.
Once we restart the nginx service the load reduces and so the response time, but same comes back in 5 min duration.
And if we stop the Nginx service on that server load decreases and same load gets shifted to other with high "Writing" value.

We have the Cache Directory Mounted on SSD as well as RAM .
We are using CentOS 6.5 with both IPV4 and IPV6 enabled on the server.
Application/User Traffic directly connect to Server over IPV4/IPV6 IP .

Output of iostat:
avg-cpu: %user %nice %system %iowait %steal %idle
0.17 0.00 0.60 0.00 0.00 99.22

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 11.75 24.82 402.35 659063678 10683772144
sdb 454.37 2180.70 3663.85 57905655098 97288688008
sdc 565.74 1771.80 4512.49 47047922584 119823249968
dm-0 0.00 0.00 0.00 5136 0
dm-1 0.05 0.40 0.30 10621106 7853376
dm-2 1040.29 3952.19 8176.34 104945070786 217111937976
dm-3 1.01 1.40 8.06 37053274 214028160
dm-4 33.97 2.39 271.65 63438578 7213330456
dm-5 15.55 20.55 122.34 545791730 3248559664

avg-cpu: %user %nice %system %iowait %steal %idle
0.58 0.00 4.52 0.02 0.00 94.88

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 181.00 0.00 34.60 0.00 1724.80 49.85 0.01 0.17 0.09 0.30
sdb 0.00 68.40 46.40 781.40 10710.40 6798.40 21.15 0.31 0.37 0.03 2.88
sdc 0.00 26.00 59.80 552.60 13425.60 4628.80 29.48 0.28 0.46 0.05 3.22
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 106.20 1428.40 24136.00 11427.20 23.17 0.79 0.52 0.04 6.00
dm-3 0.00 0.00 0.00 0.20 0.00 1.60 8.00 0.00 3.00 3.00 0.06
dm-4 0.00 0.00 0.00 212.00 0.00 1696.00 8.00 0.13 0.61 0.00 0.08
dm-5 0.00 0.00 0.00 3.40 0.00 27.20 8.00 0.00 0.47 0.47 0.16


"