Almost all of this time in the SSL handshake is probably spent on waiting for the network. But a factor of 10x seems unreasonable; I usually see 3x-4x latency increases for HTTPS compared with HTTP.

Things to test out:
1) Disable ephemeral diffie-hellman cipher suites (which real browsers don't use, but OpenSSL testing tools will, skewing your results.)
2) Use RSA+SHA where you can. Theoretically less secure than AES, but no known breaks and much faster than AES depending on hardware (key setup in particular).
3) You can't change the speed of light, so if you have a 60 ms round-trip time, SSL negotiation is going to take at least 240 ms even if client and server were infinitely fast. Test on localhost versus a remote connection to see where your bottlnecks really are
4) make sure you use the SSL session cache so you don't have to do an SSL renegotiation even if TCP connection has been ended
5) use the prefer sever ciphers feature so you control what SSL options are used
6) make sure you don't have MTU issues. SSL negotiation can generate large packets, and if you have an MTU of less than 1500 bytes, but you or clients are blocking ICMP packet too big, things will get slow as clients have to re-send smaller packets.

My SSL settings look like this:
#only use secure TLSv1 and SSLv3, not insecure SSL2
ssl_protocols TLSv1 SSLv3;
#set up preference list, disabling very slow or insecure encryption
ssl_ciphers RC4:AES128-SHA:TLSv1:SSLv3:!ADH:!aNULL:!DH:!EDH:!eNULL:!LOW:!SSLv2:!EXP:!NULL;
#use my preference list to determine encryption instead of clients
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:16m;