Strange i think you are right i access the same mp4 size and vide length on other sites and they all do the same thing take like upto a miniute before they will play but a webm will play instantly.

Any idea how i can fix this ?

But what makes it strange is it does not happen with YouTube videos that are 10-48 hours long but then i think youtube uses rtmp streams what is proborly why.