Welcome! Log In Create A New Profile

Advanced

Re: limit_req for spiders only

October 14, 2013 09:25AM
Hello.

Doesnt robots.txt "Crawl-Delay" directive satisfy your needs?
Normal spiders should obey robots.txt, if they dont - they can be banned.
SubjectAuthorPosted

limit_req for spiders only

Toni MuellerOctober 14, 2013 08:00AM

Re: limit_req for spiders only

SylviaOctober 14, 2013 09:25AM

Re: limit_req for spiders only

Toni MuellerOctober 14, 2013 10:04AM

Re: limit_req for spiders only

Francis DalyOctober 14, 2013 10:24AM

Re: limit_req for spiders only

Toni MuellerOctober 14, 2013 10:54AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 103
Record Number of Users: 5 on December 17, 2014
Record Number of Guests: 180 on December 21, 2014
Powered by nginx    Powered by FreeBSD    PHP Powered    Powered by Percona     ipv6 ready