Hi All, I am using 4 nginx servers. When I do www.example.com/robots.txt it shows Request-rate: 1/10 # maximum rate is one page every 10 seconds Crawl-delay: 10 # 10 seconds between page requests V isit-time: 0400-0845 # only visit between 04:00 and 08:45 UTC I have not configured robots.txt any where in my nginx.conf and also verified the LB setting too. I also couldn't find a fileby jithu212 - Other discussion
![]() |
![]() |
![]() |
![]() |
|