Welcome! Log In Create A New Profile

Advanced

Re: Как отбиться от 80legs.com ?

Peter B. Pokryshev
October 29, 2013 10:46AM
On Tue, 29 Oct 2013 10:40:10 -0400
"Gaidamak" <nginx-forum@nginx.us> wrote:

> Повадилась такая вот напасть.
>
> http://www.80legs.com/webcrawler.html
>
> Как ее грамотно выпилить?
>

Забанить по юзерагенту или как они на сайте сами пишут:

If you'd like us to stop crawling your website, the best thing to do is to block our web crawler using the robots.txt specification. To do this, add the following to your robots.txt:

User-agent: 008
Disallow: /
If you block 008 using robots.txt, you will see crawl requests die down gradually, rather than immediately. This happens because of our distributed architecture. Our computers only periodically receive robots.txt information for domains they are crawling.


> В логах много такого:
>
> 109.166.134.39 - - [29/Oct/2013:18:34:09 +0400] site.domain.com "GET
> /page/url/ HTTP/1.1" 502 107 "-" "Mozilla/5.0 (compatible; 008/0.85;
> http://www.80legs.com/webcrawler.html) Gecko/2008032620" 0.000
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?21,244236,244236#msg-244236
>
> _______________________________________________
> nginx-ru mailing list
> nginx-ru@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-ru

--
Peter B. Pokryshev <ppb@valuehost.ru>

_______________________________________________
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru
Subject Author Posted

Как отбиться от 80legs.com ?

Gaidamak October 29, 2013 10:40AM

Re: Как отбиться от 80legs.com ?

Peter B. Pokryshev October 29, 2013 10:46AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 154
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready