For reference I'm pretty sure that its not connection setup / negotiation that is causing the load when enabled as the machines aren't dealing with that may new connections per second, so its an overhead for the entirety of the data send I believe.

Ok so found something interesting, while mp4's are being sent with the correct header flv's aren't and hence are getting compressed:

GET /streaming/test.flv?..........

nginx/0.8.53
Date Sat, 08 Jan 2011 03:01:24 GMT
Content-Type text/html; charset=utf-8
Content-Length 693656659
Last-Modified Fri, 07 Jan 2011 16:19:25 GMT
Connection keep-alive
Cache-Control private, max-age=0, must-revalidate
Accept-Ranges bytes

GET /streaming/test.mp4?..........
Server nginx/0.8.49
Date Sat, 08 Jan 2011 03:09:25 GMT
Content-Type video/mp4; charset=utf-8
Content-Length 53385372
Last-Modified Wed, 05 Jan 2011 14:55:46 GMT
Connection keep-alive
Cache-Control private, max-age=0, must-revalidate
X-Mod-H264-Streaming version=2.2.7
Accept-Ranges bytes

So the question is now why are flv's getting text/html as their content type?