BFD is banning good search bots on this error: \\x16\\x03\\x01

[elysium26]

I keep getting an error showing up in my apache error log file that is banning good search bots. I just wanted to see if any one else has seen this problem and how they fixed it. I installed BFD following the instructions in theads for JPC-Greg and thisisit3.

Here is an example from my error log (that happened today):
[Fri Aug 17 06:13:23 2007] [error] [client 65.54.188.59] Invalid method in request \\x16\\x03\\x01
[Fri Aug 17 06:13:23 2007] [error] [client 65.54.188.59] Invalid method in request \\x16\\x03\\x01
[Fri Aug 17 06:13:24 2007] [error] [client 65.54.188.59] Invalid method in request \\x16\\x03\\x01

This IP then was banned by BFD as an apache-exploit. After looking up this IP address I see that it is a Microsoft search bot/crawler. I've seen some other search bots get banned before also, and I see this as a bad thing.

What is going on is the bots are trying to connect to my hosted sites using https when there is not SSL installed for the hosted sites.

So my question is, is there a way to fix this in Apache? I think it may be the httpd.conf file that needs correcting. (Adding the port number to the virtual host declaration.) How could I change the BFD apache-exploit rule? Any ideas?

[thisisit3]

It went back and looked at all the banned IP addresses that generated the above error and all of them were some kind of automated spam bot or vulnerability scanner, they came from Korea and other weird countries.

At least on my servers, not a single one came from a known search engine bot. For example, googlebot won't go to an https:// address unless that address is a known one or newly found, so i don't see how googlebot will go to a domain that never had a secure page.

I believe the problem isn't that your domain doesn't have https, but i believe the client bot doesn't support https itself. Giving it more chances of being a spam bot, since major bots from google/yahoo fully support https.

Unfortunately its not something that needs fixing from Apache, but if you want to stop BFD from banning these IP addresses then you need to modify the /usr/local/bfd/rules/apache-exploit script:

1) comment out or remove line 23
2) go to the next line (was line 24, now will be line 23), change the end of the line
from:
>> $TMP/.apache-exploits`
to:
> $TMP/.apache-exploits`

thats it, BFD will no longer check for "invalid method in request".