Welcome! Log In Create A New Profile

Advanced

Re: limit_req for spiders only

Previous Message Next Message
Forum List Message List New Topic Print View
Sylvia
October 14, 2013 09:25AM
Hello.

Doesnt robots.txt "Crawl-Delay" directive satisfy your needs?
Normal spiders should obey robots.txt, if they dont - they can be banned.
Reply Quote
RSS
Subject Author Posted

limit_req for spiders only

Toni Mueller October 14, 2013 08:00AM

Re: limit_req for spiders only

Sylvia October 14, 2013 09:25AM

Re: limit_req for spiders only

Toni Mueller October 14, 2013 10:04AM

Re: limit_req for spiders only

Francis Daly October 14, 2013 10:24AM

Re: limit_req for spiders only

Toni Mueller October 14, 2013 10:54AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 121
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready