I have a test setup to measure the nginx overhead when plugged in front of a Jboss tomcat server.
In tomcat I have deployed a test.jsp and use ab to measure the performance with the following scenarios

With out keep alive ab option
ab --> <jboss tomcat URL> (ab -n 5000 -c 5 http://jboss.tomcat.url/test.jsp)
ab --> <nginx URL> --> <jboss tomcat URL> (ab -n 5000 -c 5 http://nginx.proxy.url/test.jsp)

With keep alive ab option (-k)
ab --> <jboss tomcat URL> (ab -n 5000 -c 5 -k http://jboss.tomcat.url/test.jsp)
ab --> <nginx URL> --> <jboss tomcat URL> (ab -n 5000 -c 5 -k http://nginx.proxy.url/test.jsp)

The performance numbers WITHOUT keep alive is almost same BUT WITH keep alive option the performance numbers are very different and nginx takes about 5 secs more than the page accessed directly via tomcat. Tomcat takes only 0.503430 on an average

Why should there be so much of deviation with Keep alive? Is there anything I am missing?

-------------Tomcat per result using ab keep alive (ab -n 5000 -c 5 -k http://jboss.tomcat.url/test.jsp)-----------------------

Server Software: Apache-Coyote/1.1
Server Hostname: jboss.tomcat.host
Server Port: 9080

Document Path: /test.jsp
Document Length: 301 bytes

Concurrency Level: 5
Time taken for tests: 0.503430 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4952
Total transferred: 2944760 bytes
HTML transferred: 1505000 bytes
Requests per second: 9931.87 [#/sec] (mean)
Time per request: 0.503 [ms] (mean)
Time per request: 0.101 [ms] (mean, across all concurrent requests)
Transfer rate: 5710.82 [Kbytes/sec] received

-------------Nginx perf result using ab keep alive (ab -n 5000 -c 5 -k http://nginx.proxy.url/test.jsp) --------------------------

Server Software: nginx
Server Hostname: nginx.proxy.url
Server Port: 80

Document Path: /test.jsp
Document Length: 301 bytes

Concurrency Level: 5
Time taken for tests: 5.440499 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4952
Total transferred: 2654760 bytes
HTML transferred: 1505000 bytes
Requests per second: 919.03 [#/sec] (mean)
Time per request: 5.440 [ms] (mean)
Time per request: 1.088 [ms] (mean, across all concurrent requests)
Transfer rate: 476.43 [Kbytes/sec] received

-------------Tomcat per result WITHOUT ab keep alive (ab -n 5000 -c 5 http://jboss.tomcat.url/test.jsp)-----------------------

Server Software: Apache-Coyote/1.1
Server Hostname: jboss.tomcat.host
Server Port: 9080

Document Path: /test.jsp
Document Length: 301 bytes

Concurrency Level: 5
Time taken for tests: 4.658429 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 2920000 bytes
HTML transferred: 1505000 bytes
Requests per second: 1073.32 [#/sec] (mean)
Time per request: 4.658 [ms] (mean)
Time per request: 0.932 [ms] (mean, across all concurrent requests)
Transfer rate: 612.01 [Kbytes/sec] received

-------------Nginx perf result WITHOUT ab keep alive (ab -n 5000 -c 5 http://nginx.proxy.url/test.jsp) --------------------------

Server Software: nginx
Server Hostname: nginx.proxy.url
Server Port: 80

Document Path: /test.jsp
Document Length: 301 bytes

Concurrency Level: 5
Time taken for tests: 4.916966 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 2630000 bytes
HTML transferred: 1505000 bytes
Requests per second: 1016.89 [#/sec] (mean)
Time per request: 4.917 [ms] (mean)
Time per request: 0.983 [ms] (mean, across all concurrent requests)
Transfer rate: 522.27 [Kbytes/sec] received

=========Configuration=================

Tomcat maxKeepAlive connector is set to 100

Nginx keepalive_requests 100;
worker_processes 4;
worker_connections 4098;
use epoll;
multi_accept on;