Spammed by Googlebot

[Rudi]

Since a few days I'm attacked (again!) by Googlebot, about 300 meg of data transfer per day.
I use "Crawl-delay: 15" but this seems to have no effect?
Should I increase to 60 or do I have other options than banning them (I do not like to)??

[Ron]

Ensure that your URLs lead to unique data, in other words make sure that you can see each page through only 1 URL. Make sure that you don't allow both "WWW" and non-WWW URLs. Things like that. Not only will that help reduce unnecessary indexing by G, but is also good in terms of avoiding duplicate content penalties.

300MB isn't a huge amount per day if you've got a sizeable site, it's about 9GB a month. Usually, I find that googlebot does a deep index fairly infrequently, so it shouldn't be everyday anyway.

I can't recall if Googlebot obeys a crawl-delay; Even if it did, a 15 second delay would allow it to access 5,760 pages a day. If each page is about 53K, there's your 300MB. If you slow it down to 60 seconds (and it is obeying the directive) you'd reduce the number of possible pages by 75% or perhaps 75MB.

Personally, I find Google to be fairly well behaved compared to some other bots out there.

Good luck!

[Rudi]

Ensure that your URLs lead to unique data, in other words make sure that you can see each page through only 1 URL. Make sure that you don't allow both "WWW" and non-WWW URLs. Things like that. Not only will that help reduce unnecessary indexing by G, but is also good in terms of avoiding duplicate content penalties.

300MB isn't a huge amount per day if you've got a sizeable site, it's about 9GB a month. Usually, I find that googlebot does a deep index fairly infrequently, so it shouldn't be everyday anyway.

I can't recall if Googlebot obeys a crawl-delay; Even if it did, a 15 second delay would allow it to access 5,760 pages a day. If each page is about 53K, there's your 300MB. If you slow it down to 60 seconds (and it is obeying the directive) you'd reduce the number of possible pages by 75% or perhaps 75MB.

Personally, I find Google to be fairly well behaved compared to some other bots out there.

Good luck!

Thanks a lot!