Solution was stupid. I checked the error logs in nginx and found out that my relative path was not where I expected it to be.

This works:
location ~ /(admin/.*|test) {
auth_basic "Restricted";
auth_basic_user_file htpasswd;
}