I have nginx rate-limiting working when using the following

limit_req_zone $binary_remote_addr zone=mylimit:20m rate=50r/m;

I now want to apply it to certain IPs so i've changed it to

geo $limit {
default 1;
1.2.3.4/32 0;
}

map $limit $mylimit {
0 "";
1 $binary_remote_addr;
}

limit_req_zone $my_limit zone=mylimit:20m rate=50r/m;

Following the example here https://www.nginx.com/blog/rate-limiting-nginx/

But the rate limit is ignored even when coming from a different IP than the one in the config

This is using nginx version: nginx/1.14.0 (Ubuntu)

In the server block I have

limit_req zone=mylimit burst=15 nodelay;

which was working before