B.R.

They are serving exactly the same resources at the same time... My vhost points to the same folders for each domain. Files are accessible over HTTP and HTTPS. The slow down comes when downloading (the same resource) from HTTPS. For example:

http://webmail.domain.tld/test.zip (30MB file can be downloaded at 4-5MB/s)
https://webmail.domain.tld/test.zip (30MB file can be downloaded at 300-700kb/s)

I enabled SPDY to see if that would've made a difference but it didn't. I do have SPDY currently enabled.