Hello,

I'm having a weird problem with my website.
In my nginx conf, I have this rule:
rewrite ^/robots\.txt$ /cms/robotstxt.php last;

and a location to handle PHP files:
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}

When I visit robots.txt, I get the expected result (a robots.txt dynamically generated). If I check the nginx log file, as expected I get a 200 HTTP answer :
IPADDRESS - - [21/Sep/2010:16:22:57 +0200] "GET /robots.txt HTTP/1.0" 200 231 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)"
If I check with fiddler (a HTTP debugger), everything is also OK : I have my robots.txt file with the 200 HTTP code.

By checking the logs most of the time the 200 code is here.

Now, here is the problem : sometimes, for some reasons that I don't know, instead of the 200 code, nginx send a 301 code. Here's an example of the googlebot visiting my website :
66.249.65.168 - - [21/Sep/2010:12:02:09 +0200] "GET /robots.txt HTTP/1.0" 301 178 "-" "Googlebot-Image/1.0"
66.249.65.166 - - [21/Sep/2010:12:02:09 +0200] "GET /cms/robotstxt.php HTTP/1.0" 200 231 "-" "Googlebot-Image/1.0"

Of course, I don't want bots or people to visit this /cms/robotstxt.php page directly...

Tonight, for the first time I've also found this problem with another rewrite rule. Again, I have a rule like :
rewrite "^/([0-9]+)-([a-z0-9-]*)-([a-z]{2})$" "/cms/pages.php?id=$1;title=$2;language=$3" last; and normally I get a 200 HTTP code when I visit one of my page.
But I discovered some weird logs from the Yandex bot :
95.108.151.244 - - [21/Sep/2010:21:20:56 +0200] "GET /123-mypage-fr HTTP/1.0" 301 178 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)"


I've also tried to fetch the robots.txt file from the google webmaster tools, but I got a correct 200 HTTP answer.

Is anybody know what the problem could be ? I have no idea how I could reproduce this strange results with my browser or wget or whatever tool.


I'm using nginx 0.7.67, the version packaged with Debian Squeeze 32bits.

Thanks.