Thanks Maxim.

Setting either 'accept_mutex pff;' or reuseport on the listen directive doesn't seem to change the general trend much

(apologies for formatting below but can't seem to find a way to paste data elegantly in this forum)


In data below the Columns are as follows

"Wrk_threads" "Connections" "Default config" "access_log off';" "access_log off; + accept_mutex off;" "access_log off; + listen...reuseport"

Requests/sec:
8 8 19352.37 22290.17 23944.48 23448.55
16 16 24839.12 28408.74 26976.59 27545.21
16 24 24845.17 28479.84 27039.90 27631.32
16 32 24633.34 25373.32 22097.93 25792.31
16 48 24540.14 21701.73 19953.71 23836.73
16 72 24620.15 19669.01 19285.64 20222.72
16 96 24591.50 19616.52 19646.13 19669.74
16 120 24569.45 19815.90 19752.29 19818.98
16 200 24642.54 20001.03 19981.29 19974.83
16 300 24161.27 20185.21 20376.75 20381.96
16 400 24607.07 20526.47 20743.34 20459.76
16 500 24712.25 20626.14 21016.65 20927.98
16 1000 24597.04 21137.30 21377.55 21032.03

Average Latency:
8 8 0.87ms 774.84us 348.81us 580.52us
16 16 1.61ms 1.31ms 679.01us 2.14ms
16 24 1.25ms 1.06ms 642.36us 1.52ms
16 32 3.94ms 3.23ms 1.93ms 10.26ms
16 48 8.71ms 5.00ms 3.25ms 26.24ms
16 72 17.21ms 5.37ms 3.95ms 7.53ms
16 96 36.09ms 9.79ms 8.49ms 8.05ms
16 120 44.31ms 12.13ms 9.82ms 9.82ms
16 200 108.02ms 22.18ms 21.58ms 19.26ms
16 300 112.68ms 36.75ms 39.94ms 35.11ms
16 400 94.84ms 58.44ms 60.78ms 53.03ms
16 500 82.97ms 78.26ms 65.91ms 71.37ms
16 1000 50.99ms 164.28ms 156.12ms 165.86ms

Socket errors:
8 8 none none none none
16 16 none none none none
16 24 none none none none
16 32 none none none none
16 48 none none none none
16 72 none none none none
16 96 none none none none
16 120 none none none none
16 200 none none none none
16 300 timeout 35 none none none
16 400 read 14, timeout 44 none none none
16 500 read 37, timeout 111 none none timeout 57
16 1000 timeout 189 timeout 473 timeout 629 timeout 1162


I had been experimenting with various tuning parameters for both NGINX and Linux including

NGINX
worker_connections, access_log, worker_rlimit_nofile, multi_accept, keepalive_requests

Linux
somaxconn, tcp_max_tw_buckets, netdev_max_backlog, tcp_max_syn_backlog, ip_local_port_range, tcp_fin_timeout, tcp_tw_recycle, tcp_tw_reuse, wmem_max, tcp_rmem, tcp_wmem

When I was getting undesired results with 'optimiizations' - I started backing off the various tunings above - for some reason the access log off seems to be the main perpetrator - for reasons I don't understand

On the client side I'm running on a machine with Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz (4 core/ 8 thread). I've repeated the runs with both 8 and 32 wrk threads with similar trend using access_log off;