Defeating spam: comments, referrer, e-mail spambots...

[Melfina]

This post must remain as an informative article to those people having problems with spam. I'm still learning of course, so this post will be updated whenever I find something else interesting about this topic.

As far as I know, there are three main spam types:
1. E-mail spam
2. Commenting spam
3. Registration spam
4. Referrer spam
Being the first one, I think, the most known and the last one maybe the least known.

Each type of spambot commented below.
These forms of massive spam are carried by spambots, not by people.

E-mail spambot
The e-mail spambot works by browsing through sites and their links collecting e-mail adresses to send them spam.
This one's the most common and the most easy to avoid, I think. Do not ever put mailto links ( <a href="mailto:whatever@domain.com">e-mail</a> ) or write real e-mails on your pages:
- Mask them instead: user

• domain.com instead of user@domain.com or user@NOSPAMdomain.com

- Use javascript to show e-mails or mailto links: http://joemaller.com/js-mailer.shtml
- Use contact forms instead. Much better if they have a confirm page before they're sent.
- Put e-mails on an image instead of plain text.

Comment spambots
These spambots will keep spamming your commenting forms to put links to nasty websites such as poker, porn, drugs, etcetera. It won't be long until you get tired of deleting spam comments every day.
I'm still not really sure how to stop this ones, but these may work:
- Adding confirmation pages before the comments are added.
- Admin must confirm new comment entries before they're displayed. I know it's a pain, but that's another way.
- Some news script seem to be bot-safe. I never saw spam comments when I was using CuteNews, for example.
- Block their user-agents (because they are bots!!) through .htacces , adding this:

Code:

SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

This will keep spambots out of your page, saving bandwidth and server load too. Of course, you must complete this bots list by checking which bots are accessing your site (remember, there are search engine bots which are not harmful!! they keep your site listed on the main search engines such as google and yahoo) and add them to the blocked list at .htaccess

Registering spambots
They register through forums and limit the usernames available and put nasty sites on their personal website. Stop them by:
- New registered members must activate their account through e-mail
- Visual confirmation codes. This is a default protection on the new versions of phpBB, so if you're running an old version, upgrade your script and your templates.

Referrer spambots
They'll put those nasty websites on your referrer pages, though they don't really supply a link to your site. This way they get more hits. It seems we can also stom them through .htacces by stopping certain referrers:

Code:

SecFilterEngine On
SecFilterDefaultAction "deny,status:412"
SecFilterSelective "HTTP_REFERER" "(holdem|poker|loan|mortgage|hold-em)"

Warning: things that won't work
- Blocking IP adresses: their IP adresses are different everytime.
- Changing form field names: it won't be long until they find out the new field names.
- Blocking certain words does help, but it's not 100% secure, not even 60%, try alternative ways too.

Defeating Spam
The main search engine websites are reaching an agreement to start defeating spam. It seems that links coded like <a rel="nofollow"> won't be followed by bots, therefore not listed or its page rank increased. That's pretty much what the spammers are looking for when they fill your website with urls.

"[...]Fighting this web spam is one of the harder parts of our job.  Spammers are very creative people who evolve their techniques over time.  Over the past year, being the newbie to the search engine block, we have had to discover and combat all of the techniques that have been developed over the past 5 years.  Now, finally, we get to move forward![...]"

"[...]I quickly circulated this around our hallway and also to the MSN Spaces team ? and we got quick agreement from both teams.  Over the coming weeks, our MSNBot crawler will start respecting this new tag, and sometime after that MSN Spaces will start to support this as well (I expect Mike Torres and MC will have more details ? the team has already made investments to help prevent blog spam to begin with). Cheers to Yahoo! and Six Apart for also supporting this movement.[...]"


This article is not complete!! the suggested solutions may not work for you, this spambots thing is really complex, more than you may imagine. I still need to document more and complete this post.

Links to helpful articles and information
How I block comment spam
Six Part guide to Combatting Comment spam

[Hikari]

excellent article Melfina, i really needed that (garr, I was so sick of comment spam ; )

[Donna]

Wow Melfina thanks! i have a BIG problem with Spambots cuz they always send me spam mail and sign in my guestbook -.-

[Kim]

Thanks Melfina. You're the best! ^.~ But one thing that I still don't get. When I tried posting this code to my .htaccess....

Code:

SecFilterEngine On
SecFilterDefaultAction "deny,status:412"
SecFilterSelective "HTTP_REFERER" "(holdem|poker|loan|mortgage|hold-em)"

....it gives me an Internal Server Error. o_O

[Melfina]

Did it?? ?_.!! omg I'm using at and didn't get anything, though I left a line that was made by the server. My full .htaccess is this one:

Code:

<Files 403.shtml>
order allow,deny
allow from all
</Files>



SecFilterEngine On
SecFilterDefaultAction "deny,status:412"
SecFilterSelective "HTTP_REFERER" "(holdem|poker|loan|mortgage|hold-em|texas-hold|voyeur|jackpot|child-porn|gambling)"

SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

[Kim]

It probably hates me, Melfina. u_u;; Oh well, at least the first one works. What happens if I didn't put the second code on? Will there be spambots coming to my site?

[Melfina]

Maybe not. If you have cpanel, you can log in and check your referrers. There you'll see if you have referrers with urls like "texas-holdem" "poker-online" and so on. I've noticed that they aren't logged with extreme tracker anyways.

I've been trying to add a piece of code to my htaccess today and it gave me an internal server error. It seems that depending the type of server you're running on or how it's setup, you are able to run certain things on .htaccess or not ?_.! It seems mine won't let me use "FilesMatch" :S!

You can always ask your host to see what you can do with it.

[Kim]

All right. Thanks a lot Melfina! :lol:

[Shirubia]

Did it?? ?_.!! omg I'm using at and didn't get anything, though I left a line that was made by the server. My full .htaccess is this one:

Code:

<Files>
order allow,deny
allow from all
</Files>



SecFilterEngine On
SecFilterDefaultAction "deny,status:412"
SecFilterSelective "HTTP_REFERER" "(holdem|poker|loan|mortgage|hold-em|texas-hold|voyeur|jackpot|child-porn|gambling)"

SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot

<Limit>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

It gave me an internal server error when I tried it, too  :shock:  :cry:

[vini1_2]

Why can't ppl who are on free server like yahoo geo see their ht access file? Don't they have one? Ah hold on I think on its yahoo's fault, it makes pageinvalid by posting its ad on it which is invalid.. it probably hides ht access file too.

[Shirubia]

Why can't ppl who are on free server like yahoo geo see their ht access file? Don't they have one? Ah hold on I think on its yahoo's fault, it makes pageinvalid by posting its ad on it which is invalid.. it probably hides ht access file too.

Those "privileges" (like acces to your htacces file, cgi-bin, etc.) are restricted by most of the free servers. They don't offer you total control of your site like paid servers do.
Maybe it's for security reasons too.

[Skailurr]

I just simply use
http://english-135911360720.spampoison.com/
for Cutenews spambots. Thanks thought for the tutorial since it will help me in the future incase I stop using Cutenews and other stuff @_@