Welcome! Log In Create A New Profile

Advanced

Re: Allow internal redirect to URI x, but deny external request for x?

All files from this thread

File Name File Size   Posted by Date  
smime.p7s 3.5 KB open | download j94305 09/03/2019 Read message
smime.p7s 3.5 KB open | download j94305 09/04/2019 Read message
September 10, 2019 02:46PM
Robots exclusion is generally quite unreliable. Exclusions based on user agents are also not really reliable. You can try all of the options for robots exclusion and may still get undesired crawlers on your site.

The only way you can keep robots out is to require authentication for those parts you don't want to have crawled.

--j.
Subject Author Posted

Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 30, 2019 01:34PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 30, 2019 02:22PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 30, 2019 02:38PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 30, 2019 03:00PM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly August 30, 2019 04:56PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 30, 2019 06:00PM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly August 30, 2019 07:22PM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly August 31, 2019 03:28AM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 31, 2019 11:06AM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 31, 2019 10:12AM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly August 31, 2019 04:52PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 31, 2019 05:56PM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly September 02, 2019 05:04PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir September 03, 2019 01:28PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir September 03, 2019 05:32PM

Re: Allow internal redirect to URI x, but deny external request for x?

Ian Hobson August 30, 2019 03:02PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir August 31, 2019 11:32AM

Re: Allow internal redirect to URI x, but deny external request for x?

Francis Daly August 30, 2019 04:34PM

Re: Allow internal redirect to URI x, but deny external request for x?

gariac August 30, 2019 05:24PM

Re: Allow internal redirect to URI x, but deny external request for x?

Ian Hobson August 31, 2019 10:42AM

Re: Allow internal redirect to URI x, but deny external request for x?

j94305 August 30, 2019 06:28PM

Re: Allow internal redirect to URI x, but deny external request for x?

gariac August 31, 2019 03:20PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir September 03, 2019 11:30PM

Re: Allow internal redirect to URI x, but deny external request for x? Attachments

j94305 September 03, 2019 11:56PM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir September 04, 2019 11:30AM

Re: Allow internal redirect to URI x, but deny external request for x? Attachments

j94305 September 04, 2019 11:44AM

Re: Allow internal redirect to URI x, but deny external request for x?

J. Lewis Muir September 04, 2019 12:32PM

Re: Allow internal redirect to URI x, but deny external request for x?

j94305 September 10, 2019 02:46PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 74
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready