Robot Exclusion Standard

[Ron]

Does Nameprotect really think that my robots.txt file is going to change this frequently?

EDIT: to avoid a hugely wide page, I've edited these lines out. but they all refer to the same exact path and robots.txt file as this one:
[Tue Jun 8 11:09:12 2004] [error] [client 12.175.0.35] File does not exist: /home//public_html/forum/robots.txt

[Tue Jun 8 11:09:12 2004] [error] [client 12.175.0.35]
[Tue Jun 8 11:08:52 2004] [error] [client 12.175.0.35]
[Tue Jun 8 09:42:02 2004] [error] [client 12.175.0.35]
[Tue Jun 8 08:29:35 2004] [error] [client 12.175.0.35]
[Tue Jun 8 08:29:25 2004] [error] [client 12.175.0.35]
[Tue Jun 8 06:55:09 2004] [error] [client 12.175.0.35]
[Tue Jun 8 05:49:05 2004] [error] [client 12.175.0.35]
[Tue Jun 8 05:48:45 2004] [error] [client 12.175.0.35]
[Tue Jun 8 04:41:44 2004] [error] [client 12.175.0.35]
[Tue Jun 8 03:27:22 2004] [error] [client 12.175.0.35]
[Tue Jun 8 02:20:56 2004] [error] [client 12.175.0.35]
[Tue Jun 8 01:01:25 2004] [error] [client 12.175.0.35]
[Tue Jun 8 01:01:18 2004] [error] [client 12.175.0.35]
[Mon Jun 7 23:28:32 2004] [error] [client 12.175.0.35]
[Mon Jun 7 22:08:28 2004] [error] [client 12.175.0.35]
[Mon Jun 7 22:08:20 2004] [error] [client 12.175.0.35]
[Mon Jun 7 20:59:08 2004] [error] [client 12.175.0.35]
[Mon Jun 7 19:45:58 2004] [error] [client 12.175.0.35]

[Spathiphyllum]

Dang Ron,

How much content are you trying to siphon from my site? My hired-gun spider is really having to keep an eye on you.

[Ron]

I mean, really! And to boot, it's my FORUM that they're after! Not like they're going to find a copyright violation on a public board. Unless they're just trying to find out what brand of shampoo people with sheepdogs are recommending these days.

[Ron]

Frankly, I'd put up a robots.txt file, but I'm very worried that I'll make an error and accidentally exclude googlebot or some other bot that I want. Specifically but vaguely, one bot that I want to visit says that if you have a robots.txct file you MUST specifically allow their bot access in the file.

How crappy is that?

[Vin DSL]

Screw that 'robots.txt' stuff. That only works for bots that follow the rules. See if you can pull up a UA on the bot[s] and ban them from your site. Simple pimple!

[Vin DSL]

Doing a quick search on Google, I found this:

http://diveintomark.org/archives/200..._to_go_to_hell

Looks fairly comprehensive! That should give you idea what I'm talking about. If that doesn't do it for you, let me know...

[Vin DSL]

Hahahaha! Clever guys!

You might have to get to that page using this link:

http://216.239.57.104/search?q=cache...n+bot s&hl=en

[Vin DSL]

Here's another one for you:

http://www.evolt.org/article/Using_A...bots/18/15126/

[Connie]

Originally posted by Ron
Frankly, I'd put up a robots.txt file, but I'm very worried that I'll make an error and accidentally exclude googlebot or some other bot that I want. Specifically but vaguely, one bot that I want to visit says that if you have a robots.txct file you MUST specifically allow their bot access in the file.

How crappy is that?

Ron,

I thought from all your other post that you were the expert about everything. I'm surprised you are having problems with a robots.text file.

[Spathiphyllum]

Originally posted by clssam
I'm surprised you are having problems with a robots.text file.

I think it has more to do with a reticence to rely on a robots.txt file to do anything since the standard is often ignored. You can visit the sites of "good" robots and follow their guidelines to invite their robot; conversely, other less developed but nevertheless useful bots may not have a posted guide. It is difficult to tell what is a robot and what is not a robot due to the signature pattern left behind - just about anything can be declared by the proggie and trying to tame them can be tedious and time-consuming. Since the protocol is not strictly enforced, abuse is rampant as are the rulesets to contain them. A misstated rule could unnecessarily eliminate a useful bot.

[Connie]

Originally posted by Spathiphyllum
I think it has more to do with a reticence to rely on a robots.txt file to do anything since the standard is often ignored. You can visit the sites of "good" robots and follow their guidelines to invite their robot; conversely, other less developed but nevertheless useful bots may not have a posted guide. It is difficult to tell what is a robot and what is not a robot due to the signature pattern left behind - just about anything can be declared by the proggie and trying to tame them can be tedious and time-consuming. Since the protocol is not strictly enforced, abuse is rampant as are the rulesets to contain them. A misstated rule could unnecessarily eliminate a useful bot.

Actually Ron referanced "googlebot" specifically.

[Ron]

Connie,
I have no idea why you're upset with me.
If I've done something to offend you, please be assured that it was unintentional, and I offer you my sincerest apology.
-Ron.

[Spathiphyllum]

Originally posted by clssam
Actually Ron referanced "googlebot" specifically.

True, but it was the "some other bot" that caught my eye.

[Connie]

Originally posted by Ron
Connie,
I have no idea why you're upset with me.
If I've done something to offend you, please be assured that it was unintentional, and I offer you my sincerest apology.
-Ron.

Ron,

I'm not upset with you and am sorry you took it that way. Actually I have gotten you confused with someone else who has openly proclaimed how intelligent and aragonite they are.

What prompted the post was I was shocked about someone who is so intelligent asking a question that is on my level.

Hope you understand this.

If I had doubled checked your web site before making the post I probably would have kept my mouth shut, or offered what lilttle help I could.

Hope you accept the apology. I think you have a great web-site. I have a sheltie who is my fur-ball son. Teddy goes every where that my truck goes to. He doesn't know anything about sheep, but he tries to herd horses, squirls, rabbits, birds, etc. It must be that scottish breeding.

Again I hope you accept my apology.

[Ron]

Dueling apologies!

S'ok....None required.