Welcome! Log In Create A New Profile

Re: Allow internal redirect to URI x, but deny external request for x?

All files from this thread

File Name	File Size		Posted by	Date
smime.p7s	3.5 KB	open \| download	j94305	09/03/2019	Read message
smime.p7s	3.5 KB	open \| download	j94305	09/04/2019	Read message

Previous Message Next Message

Forum List Message List New Topic Print View

j94305

September 10, 2019 02:46PM

Registered: 5 years ago
Posts: 36

Robots exclusion is generally quite unreliable. Exclusions based on user agents are also not really reliable. You can try all of the options for robots exclusion and may still get undesired crawlers on your site.

The only way you can keep robots out is to require authentication for those parts you don't want to have crawled.

--j.

Reply Quote

RSS

Subject	Author	Posted
Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 30, 2019 01:34PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 30, 2019 02:22PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 30, 2019 02:38PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 30, 2019 03:00PM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	August 30, 2019 04:56PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 30, 2019 06:00PM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	August 30, 2019 07:22PM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	August 31, 2019 03:28AM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 31, 2019 11:06AM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 31, 2019 10:12AM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	August 31, 2019 04:52PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 31, 2019 05:56PM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	September 02, 2019 05:04PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	September 03, 2019 01:28PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	September 03, 2019 05:32PM
Re: Allow internal redirect to URI x, but deny external request for x?	Ian Hobson	August 30, 2019 03:02PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	August 31, 2019 11:32AM
Re: Allow internal redirect to URI x, but deny external request for x?	Francis Daly	August 30, 2019 04:34PM
Re: Allow internal redirect to URI x, but deny external request for x?	gariac	August 30, 2019 05:24PM
Re: Allow internal redirect to URI x, but deny external request for x?	Ian Hobson	August 31, 2019 10:42AM
Re: Allow internal redirect to URI x, but deny external request for x?	j94305	August 30, 2019 06:28PM
Re: Allow internal redirect to URI x, but deny external request for x?	gariac	August 31, 2019 03:20PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	September 03, 2019 11:30PM
Re: Allow internal redirect to URI x, but deny external request for x?	j94305	September 03, 2019 11:56PM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	September 04, 2019 11:30AM
Re: Allow internal redirect to URI x, but deny external request for x?	j94305	September 04, 2019 11:44AM
Re: Allow internal redirect to URI x, but deny external request for x?	J. Lewis Muir	September 04, 2019 12:32PM
Re: Allow internal redirect to URI x, but deny external request for x?	j94305	September 10, 2019 02:46PM

Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 213

Record Number of Users: 8 on April 13, 2023

Record Number of Guests: 421 on December 02, 2018