My company is a heavy user of varnish cache since 2006. We have 6 varnish servers with 550 GB of RAM caching static pictures for our community. At peak each server is handling 40k active connections per second.

If you only have one server you should use a dedicated static server and let the file system cache(linux caches last read data in ram), but it won't scale beyond that server. But I assume you have many GBs of mp3s and need more than one cache. Our setup have pools of varnish servers delivering photos from RAM, and we distribute traffic between the varnishes by hash method so one server only serve his part of the photos and caches them in memory.

Varnish is also a very hardware friendly software. It makes smokeping look nice. :)