Need a larger vps? Lol
If it's virtuozzo (perhaps Xen though I don't have a lot of experience that
that) your going to hit open file limits put on the instance as well. Even
with sophisticated software based firewalls you can hit arbitrary iptable
entry limits as well.

csf has a portflood feature that works fairly well that uses the ipt_recent
module
or there are connection based monitors such as dos-deflate for firewalling
ip's that reach connection thresholds.

Though really you connection limit was probably the biggest issue. I could
see nginx handling simple requests like that no sweat if you have caching
enabled.