Combating 'referer' Spam...

[Vin DSL]

I don't know if any of you are having this problem, but I've been getting a lot of 'referer' spam lately. I'm gonna give this a try in .htaccess:

# combat referer spam
SetEnvIfNoCase Referer "^http://www.andrewsaluk.com" spam_ref=1
SetEnvIfNoCase Referer ".*(-).*(-).*"  spam_ref=1
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=spam_ref
</FilesMatch>

This 'andrewsaluk.com' character hit my server 1800+ times yesterday...

I'm also getting a lot of XXX sites with hyphens in the name (like http://free-xxx-lesbian-pics.com et cetera) The code above should block referers with at least two '-'s.

I guess the idea is for them to fool the search bots into indexing their sites. They spoof a link on a site like mine, with a Google PR of 5, or more, and get a free ride...

Anyway, I'll let you know how it works, and we'll go from there.

Any other ideas

[Vin DSL]

Oops! That worked too good!

I found out I have a file with two hyphens in the name...

[Vin DSL]

Heh! I have a feeling I'm talking to myself here?

I went through my logs and had to take a different tact with the code.

Any interest in this sort stuff?

[Ron]

Your post is a bit confusing, Vin, when you say "they spoof a link on a site like [yours]" did you mean they post a link to their site in your phpBB forum?

If they are spoofing on your site, I can only imagine that means they post a link to one site, which immediately redirects to another.. in which case your double-hyphen trick would be useless (unless the first also had double-hyphens).

If you just meant "posting a link on a site like yours" there are mods available at phpBB to prevent people from:
a) registering with a website in their profile
b) registering with a bot (using a CAPTCHA-type picture for visual confirmation)
c) posting links of any type
d) adding the <rel="nofollow"> to all user-supplied links (will stop them from getting google credit for the link, so why bother)

On the same day you posted this, I got hit with multiple posts for a lesbian site -- before that, I hadn't had a spam post in weeks. Very strange indeed. We have both been hit by very similar attacks on the exact same day. I wonder if this site is the source of our being indexed together. My SPAMmer was a single entity using the same anonymous guest username each time, and coming from the same IP address each time. It was an easy problem to combat.

Anyway, I am eternally vigilant for SPAM posts and registrations.

[Connie]

I think he is referring to links in the log files. I have several of them every month. A spider hits your site and your refer logs shows a link back to the originating site. I've never understood how this actually benefits them, since our log files are not in the public html folder.

[Ron]

Ahhh yes, I see, I missed the first line of his post I guess. Thanks for straightening me out.

I have gotten that too, but so far not from porn sites. Referrer SPAM is a very handy tool.

I don't know but maybe vin's site or php-nuke has publicly available stats?

[Vin DSL]

If you don't mind a copy 'n' paste...

Why go to the effort of leaving a web site address in someone else's log files?

Most web servers have the ability to log an extensive amount of information about web sites visitors. Many webmasters and bloggers use web-based software to parse those log files automatically. The result is one or more pages breaking the information down into very detailed statistics. These statistics include the referer information and often those referers are displayed as hyperlinks.

Bloggers quite often will display a link to the most frequent or most recent web site found in their referer logs using scripts. Some will even put those links right on the front page of their site in a sidebar area (e.g. me - VinDSL).

Unscrupulous web site owners are spamming the log files in order to have their web sites listed on those referer links. This creates an artificial boost in that site's popularity among those search engines that measure the number of links to a site. It also generates traffic when curious visitors of a victim site clicks the links displayed in the referer listing.

Simply put, these people are running advertisements on your web site and using it to boost their search engine rankings. They do this without your knowledge, without your permission, and without compensating you in any way for the use of your network.

SOURCE: http://www.spywareinfo.com/articles/referer_spam/

I actually don't have a problem with this, unless they are chewing up my bandwidth, advertising porn sites, or using my site for 'browser hijacking'.

I might also mention, this 'referer spamming' makes your AWStats, and so forth, basically useless for doing any meaningful trend analysis.

To give you an idea of 'where-I'm-coming-from', this is what I'm using now...

# Combat referer spam
RewriteCond %{HTTP_REFERER} ^http://[a-z.-]+.info/.*$               [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://[a-z.-]+.biz/.*$                [NC,OR]

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)adult(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)amateur(-|.).*$  [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)busty(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)casino(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)dating(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)discount(-|.).*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)escort(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)****(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)gay(-|.).*$      [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)hotel(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)incest(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)lesbian(-|.).*$  [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)loan(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)locator(-|.).*$  [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)mature(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)mortgage(-|.).*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)nude(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)poker(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)porn(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)pussy(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)rental(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)sex(-|.).*$      [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)teen(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)tits(-|.).*$     [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)vegas(-|.).*$    [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)viagra(-|.).*$   [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)xxx(-|.).*$      [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)4u(-|.).*$       [NC,OR]

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*accepted.cc$          [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*barnevakten.no$       [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*forskning.no$         [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*hellclan.com.hk$      [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*hot.ee$               [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*livenet.pl$           [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*roxtet.com$           [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*saab.de$              [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*shape.de$             [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*skip.pl$              [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*startkabel.nl$        [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*super-illu.de$        [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*tonspion.de$          [NC,OR]

RewriteCond %{HTTP_REFERER} ^http://(www\.)?andrewsaluk.*$          [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?blogspot.*$             [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?ducoon.*$               [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?landsend.*$             [NC]

RewriteCond %{HTTP_REFERER} (kylos)                                 [NC]
RewriteRule ^.*$ http://www.goawayanddontcomeback.com               [L]

[Spathiphyllum]

I've never understood how this actually benefits them, since our log files are not in the public html folder.

This depends on the webhost and the maintainer, clssam.

Different hosts put those logs in different places, some of which are accessible by spiders. They do this so that less web savvy webmasters may view the raw logs from a web interface. A misconfiguration here or there can open up access to these logs to robot parsing not to mention allowing symlinks and other such technical tweaks that some do not understand how to implement completely.

A log file with IPs or recorded URL referers will get scanned and indexed just like any web page. This is just a less known way to spam and create more traffic to one's site and, thus, get bumped up higher in search engine rankings. It's insidiously clever and capitalizes on those that do not run a tight web site.

[Vin DSL]

WooHoo!

I added a couple of lines of code, last night, and it seems to have taken care of 100% of my 'refer spam'. I haven't had a bogus referer in 12 hours! Every referer I've checked contains a real link to my site. Amazing!

It seems too good to be true, but the proof is in the pudding, you know? Tonight I'll comment out the other lines (above) and see what happens.

I'm sure that I'll still need to ban porn sites and other web sites with objectionable domain names, et cetera, but this code evidently addresses the #1 tactic used by 'referer spammers', e.g. spam bots...

Fingers crossed...

[lokki]

Nicely done, Vin. I appreciate you 'blogging' these sorts of things - the information is very handy, along with your solutions. Thanks!

[Ron]

IPs aren't possessive, sheesh

ISPs aren't... or are the'y?

[Vin DSL]

ISPs aren't... or are the'y?

Technically speaking, abbreviations like 'ISP' shouldn't be formed with upper-case letters either, since none of the words are proper nouns, including the word internet. The plural of 'ISP' is isps, but that looks just plain stupid... that's why ppl cap 'em and add apostrophes, not because they're possessed...

[Ron]

Hmmm , I'm not an English major, but I'd say they were technically acronyms, not abbreviations.

I beleive it's appropriate to capitalize acronyms, but that's without the benefit of a copy of Strunk and White

Here's an interesting take on the capitalization of acronyms issue:
http://www.acronymsearch.com/FAQ_003.htm

[Connie]

This depends on the web host and the maintainer, clssam.

Different hosts put those logs in different places, some of which are accessible by spiders.

Our log files are above the root directory, so spiders can't crawl them, right or wrong? If I'm wrong I'm going to get concerned about it. There is a lot of evidence right now that Google in particular is clamping down on sites linking to bad neighbor hoods.

I only use AWStats for general information so I'm not as concerned about the results slightly altered.

Vin am I correct that a copy and past of the rewrite rules into the .htaccess file you posted will take care of a lot of it.

[jason]

Connie, the JPC setup won't allow spiders to get access to your logs. The only way they'd be able to get any of that info is, sy, you installed your own stats program and didn't password protect the directory or, like Vin, you posted your "latest visitors" list on your site somewhere. If you don't do either of those then you won't have and adverse effects of referer spam so far as search rankings are concerned, although it is possible that you will still be spammed and that could mess up your statistic reports (that only you can see).

--Jason