Robots txt file question

[paradox]

I have a question about the robots txt file and multihosted domains.
If I have a multihosted domain at http://abcd.mymaindomain.com and the servers for www.abcd.com point to http://.abcd.mymaindomain.com how do I use the robot's txt file to disallow indexing of that sub domain/directory?

I know that the following would disallow the index from the front door of my main website ie www.mymaindomain.com

User-agent: *
Disallow: /abcd/

thus stopping the robots from visiting the multi hosted domain via www.mymaindomain.com/abcd/

But what if a robot entered the multihosted sub domain/directory from www.abcd.com ?

Should I put a disallow robots txt such as:

User-agent: *
Disallow: /

into the directory of www.mymaindomain/abcd/ and treat it as if it were a regular non multihosted domain?

Do robots read robots txt files in subdirectories or just the main domain directory?

Hope I made this question understandable.

Thanks for any advice.

[Eric Pauker]

Before you waste a lot of your time trying to get this to work, you should know that most search engine robots ignore things like the robots txt file and the noindex metatag in an HTML file. I set up a site last year that used all the various techniques I could find to tell robots to not index a file or directory, but a month after I put the site up, EVERY single page in EVERY directory was listed in Google, Yahoo, AltaVista, and ODP.

[CeleronXL]

Originally posted by Eric Pauker
Before you waste a lot of your time trying to get this to work, you should know that most search engine robots ignore things like the robots txt file and the noindex metatag in an HTML file. I set up a site last year that used all the various techniques I could find to tell robots to not index a file or directory, but a month after I put the site up, EVERY single page in EVERY directory was listed in Google, Yahoo, AltaVista, and ODP.

Then you didn't do it correctly.

Any `official` crawler will abide by robots.txt files.

[dwannall]

Originally posted by Eric Pauker
I set up a site last year that used all the various techniques I could find to tell robots to not index a file or directory, but a month after I put the site up, EVERY single page in EVERY directory was listed in Google, Yahoo, AltaVista, and ODP.

And to think, people spend 1000's of $$ to get listed.

[jason]

The crawlers looks for a robots.txt file in the root of a site, so placing one in the subdirectory of the multihosted domain and/or subdomain should prevent that site from being listed.

--Jason

[paradox]

Thanks everyone. I'll put a robots file in both the main domain and the mutihosted sub directory.