Hi All,

I am using 4 nginx servers. When I do www.example.com/robots.txt it shows

Request-rate: 1/10 # maximum rate is one page every 10 seconds
Crawl-delay: 10 # 10 seconds between page requests V
isit-time: 0400-0845 # only visit between 04:00 and 08:45 UTC

I have not configured robots.txt any where in my nginx.conf and also verified the LB setting too.

I also couldn't find a file named robots.txt any where in my Linux machine where nginx was installed.

Could someone help me to understand from where the above values fetched in nginx. Its really urgent?

Thank you,

Sreejith