I have a news site which is run by 4 tornado instances and nginx as reverse proxy in front of them.

Pages are rendered and cached in memcached so generally the response time is less than 3 ms according to tornado logs.

[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.43ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 3.41ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 1.96ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.48ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 4.09ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.43ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.49ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.25ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.39ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.93ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.70ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.08ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.72ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.02ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.70ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.74ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.85ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.60ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.83ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.65ms

When I test this site with ab at concurrency level 1000 I get response times around 0.8 seconds. Here is the benchmark result:

Document Length: 12036 bytes

Concurrency Level: 1000
Time taken for tests: 7.974 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 10000
Total transferred: 122339941 bytes
HTML transferred: 120549941 bytes
Requests per second: 1254.07 [#/sec] (mean)
Time per request: 797.407 [ms] (mean)
Time per request: 0.797 [ms] (mean, across all concurrent requests)
Transfer rate: 14982.65 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 7 20.8 0 86
Processing: 57 508 473.9 315 7014
Waiting: 57 508 473.9 315 7014
Total: 143 515 471.5 321 7014

Percentage of the requests served within a certain time (ms)
50% 321
66% 371
75% 455
80% 497
90% 1306
95% 1354
98% 1405
99% 3009
100% 7014 (longest request)

I can handle ~1200 requests/seconds with 1000 concurrent connections and when I do the same benchmark with 100 concurrent connections I can again handle around 1200 requests/second but response time drops to ~80 ms.

When it comes to real life with 1000 concurrent connections users will face 0.8 seconds response time which I think is a bad value.

My question is why response times increase when concurrency level is increased?