Previous Message
Next Message
Forum List
Message List
New Topic
Print View
Sylvia
October 14, 2013 09:25AM
Registered: 12 years ago
Posts: 29
Hello.
Doesnt robots.txt "Crawl-Delay" directive satisfy your needs?
Normal spiders should obey robots.txt, if they dont - they can be banned.
Reply
Quote
RSS
Subject
Author
Posted
limit_req for spiders only
Toni Mueller
October 14, 2013 08:00AM
Re: limit_req for spiders only
Sylvia
October 14, 2013 09:25AM
Re: limit_req for spiders only
Toni Mueller
October 14, 2013 10:04AM
Re: limit_req for spiders only
Francis Daly
October 14, 2013 10:24AM
Re: limit_req for spiders only
Toni Mueller
October 14, 2013 10:54AM
Sorry, only registered users may post in this forum.
Click here to login
Online Users
Guests: 249
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018