I have also experienced the same problem
system: centos 5.3 nginx:0.8.9 openssl:0.9.8K

[emerg]: SSL_CTX_set_tlsext_servername_callback() failed (SSL:)

ldd /usr/sbin/nginx:
linux-gate.so.1 => (0x00771000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00f5c000)
libpcre.so.0 => /usr/lib/libpcre.so.0 (0x00d2c000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x0059b000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x002e0000)
libdl.so.2 => /lib/libdl.so.2 (0x0086b000)
libz.so.1 => /usr/lib/libz.so.1 (0x00276000)
libc.so.6 => /lib/libc.so.6 (0x00110000)
/lib/ld-linux.so.2 (0x00fe3000)

ldd `which openssl`:
linux-gate.so.1 => (0x00d89000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x0057f000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00110000)
libdl.so.2 => /lib/libdl.so.2 (0x00659000)
libc.so.6 => /lib/libc.so.6 (0x00391000)
libz.so.1 => /usr/lib/libz.so.1 (0x0025e000)
/lib/ld-linux.so.2 (0x00375000)