So this is one of those issues it is most likely a bad configuration but my robots.txt file is returning a 404 because of another location because I am disallowing people to access any text files but I do want to allow only the robots.txt to be accessed.

location /robots.txt {
root 'location/to/robots/txt/file';
}
#This is to stop people digging into any directories looking for files that only PHP etc should read or serve
location ~* \.(txt|ini|xml|zip|rar)$ {
return 404;
}

I have not tested but I believe I could fix the above by changing my config to this.
location /robots.txt {
allow all;
root 'location/to/robots/txt/file';
}
#This is to stop people digging into any directories looking for files that only PHP etc should read or serve
location ~* \.(txt|ini|xml|zip|rar)$ {
deny all;
}

Is that the only possible way to deny access to all text files apart from the robots.txt ?

http://www.networkflare.com/