I've had the same issues and did some testing.

The following causes the issue where the SSL certificate that is defined in the default_server block is being sent for requests that end up in another server block that has a different ssl_certificate defined. This only happens when adding the IP address as server_name.

Example of issue:
server {
listen 443 default_server ssl;
server_name _;
ssl_certificate /usr/local/nginx/conf/ssl/default.crt;
ssl_certificate_key /usr/local/nginx/conf/ssl/default.key;
location / { return 403; }
}

server {
listen 443 ssl;
server_name 1.2.3.4;
ssl_certificate /usr/local/nginx/conf/ssl/1.2.3.4.crt;
ssl_certificate_key /usr/local/nginx/conf/ssl/1.2.3.4.key;
location /test { return 401;}
}

When I access https://1.2.3.4/test , I receive a 401 error as expected, but the SSL certificate being sent is the one defined in default.crt

Working:
<Same default server block>

server {
listen 443 ssl;
server_name test.hostname.com;
ssl_certificate /usr/local/nginx/conf/ssl/1.2.3.4.crt;
ssl_certificate_key /usr/local/nginx/conf/ssl/1.2.3.4.key;
location /test { return 401;}
}

Now when accessing test.hostname.com which is an A record to 1.2.3.4 , I get served the correct certificate as defined in 1.2.3.4 -- I've tested this multiple times on Ubuntu 12.04 w/ nginx as configured:
nginx version: nginx/1.2.3
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --with-http_ssl_module --user=nobody --group=nobody

Can anybody test and confirm this besides us?