Search Engines

[Autumn]

Does anyone know of a special code I can insert on my site to stop search engines from loading me into their database? Ex; google, aol, etc. etc. Thanks!

[dkadave]

I thinnk, there's a meta tag.

Something about no-index!

I'll do a search for you and get back to you!

[dkadave]

Found!

<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

[CeleronXL]

Or place a file called robots.txt in the directory you want to disallow, and put

Useragent: googlebot
Disallow: /

Or something like that.. I don't think that's the right syntax at all, and I'm too lazy to search.

[Vin DSL]

Originally posted by Autumn
Does anyone know of a special code I can insert on my site to stop search engines from loading me into their database? Ex; google, aol, etc. etc. Thanks!

Here you go --- use this:

http://submission.discoveryvip.com/metatag.htm

[Autumn]

Thanks! I hope it will work

[krazyaaron]

Originally posted by dkadave
Found!

<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

Does that block all search engines? And do you have to put that on each page, only index, ?

[dkadave]

It does all!

[jason]

Originally posted by dkadave
It does all!

But you need to put it on all pages...

--Jason

[Vin DSL]

Originally posted by jason
But you need to put it on all pages...

Yeah, you need to put them on all your pages, if you want to do it right. I don't know about your site, but looking at my stats --- probably 50% of the search engines enter through inner pages. If they only enter your site through the main page, then you don't have anything to worry about, but I think this senario would be extremely unlikely...

[dkadave]

Oh, sorry. Like Vin said.

It has to be on all pages!

[baku]

How about if I needed to prevent robots visiting only certain pages with sensitive contents ,
like "thankyou" pages with login information ? It should not be passport protected but it should not appear in the Google search results.

[wgiese]

To prevent the crawling of your entire site by well-behaved robots, you only need a single robots.txt in your root directory with the following two lines in it:

User-agent: *
Disallow: /

The above will prevent all of the major search engines from crawling your site.

I'm not doubting Vin's observations, but if you are still getting files crawled with the above in place, the robots are not well-behaved and will likely ignore any META tags you have in place.

More info HERE.

[the_ancient]

How about if I needed to prevent robots visiting only certain pages with sensitive contents ,
like "thankyou" pages with login information ? It should not be passport protected but it should not appear in the Google search results.

then add the Meta Tag to only these pages.