Thanks everyone for being so helpful. I've ended up applying Igor's suggestion.
But I've dropped this line as I wasn't sure what to do with it:
"~(?i)(Purebot|Lipperhey|MaMaCaSpEr|libwww-perl|Mail.Ru|gold crawler)" 1;

I am guessing it can be used if I want to match more then just google's user agent. But in any case what I did worked very nice.
I tested it using Firefox user agent and I got forbidden page, then tried adding my ip to geo bit and I was allowed.

And there is lot less "google spiders" on our site now, and all of them have ips I added to allowed list.

Thanks again for fast and great advices.

Since I am a noob when it comes to Nginx and now that I know I can get some help here I am sure I'll have more questions :)