Robots.txt and subdomains

[opaque]

I have shared hosting and am using some subdomains in addition to my main domain.I've noticed some search engine results are showing the url as http://www.maindomain/subdomain

so it's obvious the site is a subdomain of my main domain, when it really should be showing as http://www.subdomain.com

I still want http://www.subdomain to be searched though, not blocked.

If I make robots.txt in the main directory be:
User-Agent: *
Disallow: /subdomain

And a then add a robots.txt in the subdirectory with:
User-Agent: *
Allow: /

Would that prevent results showing up as from http://www.maindomain/subdomain but would show any http://www.subdomain.com results?

Or would the robots.txt in the main directory override and prevent anything in the subdomain being indexed?

[Gwaihir]

That's generally refered to as a multi-hosted domain.

If you don't do something about it, it will be accessible to search engines and visitors under:
servername.nocdirect.com/~accountname/multihosteddomain
server-ip/~accountname/multihosteddomain
www.multihosteddomain.com
www.maindomain.com/multihosteddomain
and, if I'm not mistaken, also under
multihosteddomain.maindomain.com

The best way around this is, AFAIK, to set up a 301 redirect in a .htaccess file in the subdirectory (the main folder of the multi hosted domain). This takes care of all vistors, bot or otherwise, avoids spreading ranking over what might be taken for "mirrors", etc.

You can also make a rule like that for the main domain, as, as you have probably realized now, is also accessible in more than one way.

[opaque]

Did some researching around on 301s and ended up using the Redirect section of CPanel to change the .htaccess, now when you click on the results they now end up at the right place, showing it as http://www.subdomain.com.

As expected, the search results are still showing http://www.maindomain/subdomain - having the 301s setup will cause this to correct itself eventually, right? Any idea of the usual timeframe for that - few days? several months? no way to know?

[Ron]

Be glad something is showing up somewhere!!!

It can take quite a while, or it can be fast. Google seems to have gotten better at it in the last year or so.

[jason]

I have a similar issue at work right now. I manage the web hosting environment for a university. We recently upgraded our environment and switched all of our URLs from a cryptic /~123www system (where 123 is a department number) that was established in the early 90's when the web was more of a fad to something that actually makes sense--like /college_name/department_name. In the course of doing this it was decided that, to make people conscious of the change and to get them to stop using their old URLs, instead of direct redirects from old the URLs, people would be directed to a redirect page with an explanation and a an updated link. This system seems to be working pretty well for us--hits to to the redirect pages from internal referrers are dropping off pretty nicely, but my tracking database is showing hundreds of hits to them referred by Google each day. So today I added code to our system sniff out GoogleBot and give it a 301 redirect instead of a redirect page. Now its just time to sit back and wait for Google to update its index.

--Jason

[Gwaihir]

no way to know?

I'll take this option