Welcome! Log In Create A New Profile

Advanced

Re: rewrite question

Richard Stanway via nginx
June 11, 2018 06:02AM
This is almost certainly not Google as they obey robots.txt. The & to &
conversion is another sign of a poor quality crawler. Check the RDNS and
you will find it's probably some IP faking Google UA, I suggest blocking at
network level.

On Fri, Jun 8, 2018 at 1:57 AM shiz <nginx-forum@forum.nginx.org> wrote:

> Hi,
>
> Recently, Google has started spidering my website and in addition to normal
> pages, appended "&amp" to all urls, even the pages excluded by robots.txt
>
> e.g. page.php?page=aaa -> page.php?page=aaa&amp
>
> Any idea how to redirect/rewrite this?
>
> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,280093,280093#msg-280093
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Rewrite question

Axel June 03, 2010 11:56AM

Re: Rewrite question

iberkner June 03, 2010 12:04PM

Re: Rewrite question

Maxim Dounin June 03, 2010 12:18PM

Re: Rewrite question

Axel June 03, 2010 12:20PM

Re: rewrite question

Richard Stanway via nginx June 11, 2018 06:02AM

Re: rewrite question

Francis Daly June 11, 2018 11:08AM

Re: rewrite question

Richard Stanway via nginx June 11, 2018 11:40AM

Re: rewrite question

shiz June 12, 2018 06:08AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 85
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready