Here are some “tells” that I have identified after analyzing my phpBB2 honey pot board with one month of spammer data.

Time Zone

A while back someone posted a MOD at phpbb.com that banned anyone that registered with the time zone of GMT – 12. If you check, you’ll find that GMT – 12 is in the middle of the ocean. Which reminds me of an old joke which I will paraphrase here:

Question: What do you call 10,000 spammers at the bottom of the ocean?
Answer: A good start!

Okay, maybe that one was just for me… on with the program…

So here are the statistics from my honey pot board for all users other than the original admin (that would be me) and the Anonymous user:

+---------------+----------+
| user_timezone | count(*) |
+---------------+----------+
|        -12.00 |      463 |
+---------------+----------+

Hm. That looks like a fairly significant tell to me. Every single spammer registered with the same time zone. Why do you suppose that is happening? Is it because that’s the default time zone? In fact, it’s not. On my honey pot board I set the board timezone to GMT – 5 which becomes the default for new user registrations. That means that spammer bots are specifically changing the time zone from -5 to -12 during their registration process. The only thing significant about -12 is that it’s the first option on the drop-down list. It would seem that the registration bots are making sure they select something, and in this case it’s something that nobody should really be selecting.

Is this a bullet-proof tell? It’s hard to know for sure, but the odds seem favorable.

User Location

What about the user location field, are there any patterns there? Here are the top 10 locations provided by spammer registrations on my board:

+-------------+----------+
| user_from   | count(*) |
+-------------+----------+
| Sex Relaxxx |      141 |
| USA         |       36 |
| Russia      |       33 |
| adult       |       19 |
| US          |       18 |
| Canada      |       18 |
| Greece      |        8 |
|             |        6 |
| Jamaica     |        6 |
| Kazakhstan  |        5 |
+-------------+----------+

The first one seems to indicate a spammer, as does the fourth. It’s hard to say much about the others.

Then there are those that enter a complete web site in the location field. There are only 6 (out of 464) on my honey pot board that did this, and to be honest I have seen legitimate users do this as well, so it would be hard to classify this as a solid tell of a spammer.

Profile Websites

For many years I observed spammers that would try to register on my boards only to get their web sites listed in their profile, which would then be displayed as a link on the memberlist. The first anti-spam measure I took was to prevent inactive members from showing up (a very simple, common, and popular MOD that can be found at phpbb.com as well). The next step was to prevent a user from entering a web site until they had posted a few times.

However, things seem to have changed. These simple measures became so popular that I suspect spammers started doing things to work around them. One of the changes made, interestingly enough, involved putting a legitimate website into their profile. Would you believe that one of the most popular web site entered by spammers now is google? Now I like to blame google for lots of things, but I doubt that they’re really behind all of the spammers joining my board.

Email Address

I have had plenty of posts where I called out specific email domains being used by spammers. I think it’s relatively easy to see patterns here. For example, these are the top 10 email domains used to register on my honey pot board:

+----------------------+----------+
| email_domain         | count(*) |
+----------------------+----------+
| serpdomains.com      |      142 |
| mail.ru              |      126 |
| gmail.com            |       33 |
| gawab.com            |       28 |
| dp-blog.com          |       25 |
| mymail-in.net        |       15 |
| gmx.us               |       15 |
| greatfreemail.net    |       12 |
| mp3bank.in           |        9 |
| paydayloancourse.com |        4 |
+----------------------+----------+

Notice who is number three on the list? That’s right, gmail. Along with spammer favorites like mail.ru and gawab.com I now have to deal with spammers using gmail accounts. It’s relatively easy to justify banning an email domain like anotherstupeddomain4bots.org (yes, I really got that, along with other domains in this post). I have heard of board owners that take the rather drastic step of banning all “free” email providers including hotmail and yahoo. I don’t think that’s a good step to take if you are trying to attract a wide range of members. Based on behavior I don’t have any problem adding certain domains to my banlist. I do have a problem with banning gmail and other free email accounts just because some spammers use their service.

Conclusion

Are any of these individual “tells” enough to block spammers? Maybe. Certain fields seem to have a higher success rate (time zone, for example) at predicting whether an account was created by a spammer or not. The problem with relying on an individual field like time zone is that it would be easy for a bot writer to change that behavior. In addition to that, I can’t be 100% sure that it’s not a legitimate user. For example, I just checked my biggest board and I have 21 users (a whopping 0.06%) that registered with the -12 time zone. Most of them have posted at least once and have survived, so they’re not spammers. If they were, I would have figured that out by now. In my opinion that means that I can’t really “auto-ban” anyone with that time zone, as attractive as that seemed at the beginning of this post.

Instead I have to look at patterns of behavior and combinations of fields. I can do that myself, or I can wait (impatiently! ) for the formal relaunch of the bbProtection service. The primary advantage of the bbProtection design is that it captures data from every subscriber and uses it to detect patterns from a much broader range of activity than any single board owner is likely to be able to do.

This post concentrated on reviewing registration data. Are there patterns in posting behavior that I can identify? It turns out the answer is “Yes”, and that there are some sobering statistics that show just how deep and wide the spammer-bot problems go.

But the first domain… it’s really in bad shape.

Spam User Registrations

User registrations are on the rise, meaning I’m getting more users registering every day. The average users per day (as reported by the Admin panel) is just under 15 right now. The first chart below shows the daily count of new users, and the following chart shows the cumulative count. They both look bad, don’t they?

First, the daily user registration count. I got a bump about a week ago and have seen more than double-digit user registrations for most of September.

The total user count has a disturbing upward trend.

Spam Post Statistics

The posting activity looks just as bad. For some reason, there was a major dip yesterday, but the total number of posts is climbing quite nicely.

First, the posting activity per day. The high point was on September 6 when I got 970 posts in one 24 hour period. The last data point was for September 12 when the posting activity dropped to “only” 187 spam posts for the day.

There is a slight dip in the cumulative chart shown below due to the relative slow day yesterday, but the overall trend is not good. Remember that so far each and every one of these posts is a spammer. There are even a few spammers that have replied to other spammer topics in the last few days. Given that I have a cron job that moves every topic into a hidden forum every ten minutes, a spammer has to work fairly quickly to reply to a topic before it’s moved out of public view.

Conclusion

Back to the disclaimer from the first paragraph. I think it’s important to reiterate that I set up this “bait” board (also known as a honey pot) on a domain that had previously been discovered by spammers. The length of time it took the spammers to find it (two days) was probably a direct result of that action. The other honey pot I set up two weeks ago has yet to receive a single registration, spammer or otherwise.

But once a board gets discovered, it can obviously be overrun quite quickly. I don’t think that’s news. Fortunately there are some easy steps that can be taken (the RAC MOD for one, my own Checkbox Challenge is another) to protect boards from spammer registrations. And hopefully soon we’ll see the return of the bbProtection service. That service will make use of patterns on boards across the web, so the data I’m collecting right now should prove to be useful. I am going to post a few observations on that in my next post.

How Much Is My Time Worth?

Let me start by quantifying how much I think my time is worth, based on a couple of easy metrics. When I do work in real life my billing rate is … well, I don’t want to post it, but it’s not cheap. I don’t think I can use that billing rate as the time value for this discussion, so let me move on. When I do work for phpBB clients, I charge between $25 and $50 an hour for the work. Yes, I am more expensive than many, but the higher rate accomplishes two things. First, it makes sure that people are serious about hiring me and not wasting my time. (There’s a theme here, see?) Second, if I can’t make at least that much working for someone else, I would rather work for myself. I have dozens of projects in various degrees of completion, and I would love to have some of them move forward. Working for someone else at $10 an hour doesn’t move my own projects forward, and I would rather work for myself for “free” that work for someone else at that lower rate.

As a result, I’m going to use $35 / hour as a nice middle-of-the-road rate for the metrics for this post.

How Much Time Is Wasted?

I will start by referencing statistics from this specific blog. Regular readers will be familiar with the process used to develop the Checkbox Challenge MOD. It was first put in place to block spam comments for this blog. Before that code was in place, I did use Akismet… in fact I continue to use it today. Unfortunately prior to the Checkbox Challenge MOD I would have hundreds of comments in my Akisment queue every single day. I never wanted to lose any legitimate content, so I would have to spend time reviewing every single comment in the quarantine list.

Let me be fairly conservative and suggest that it takes 15 seconds to process each comment. That includes reviewing it, marking it, clicking various buttons, and so on. That means every comment blocked before it gets to Akismet saves me fifteen seconds of time. How much is that worth?

Quantify the Benefit

Here are the statistics from this blog since August of 2007. As shown here, there have been a substantial number of attempted comments.

+-----------+------------------+
| log_month | comment_attempts |
+-----------+------------------+
| 2007-08   |             3567 |
| 2007-09   |             4715 |
| 2007-10   |             4643 |
| 2007-11   |             8484 |
| 2007-12   |             3977 |
| 2008-01   |             7116 |
| 2008-02   |             7979 |
| 2008-03   |             9905 |
| 2008-04   |             6245 |
| 2008-05   |             6588 |
| 2008-06   |             7480 |
| 2008-07   |             9173 |
| 2008-08   |            10791 |
| 2008-09   |              222 |
+-----------+------------------+

There was a total of 90,885 comments attempted. How many of them were successful? A quick check revealed that out of all of those attempts only 274 comments were successully processed. That means that 90,611 were blocked before they ever got to Akismet. If I had spent 15 seconds reviewing each of those, the total time comes out to 1359165 seconds. I can divide that number by 60 to get 22652.75 minutes, divide by 60 again to get 377.5 hours. Finally, I will multiply that number by $35 an hour and the Checkbox Challenge MOD has saved me $13,212.50.

Think about that for a minute.

Over thirteen thousand dollars in potential lost productivity due to spammer activity on this one blog.

Some might consider that 15 seconds per each comment might be too high, and that’s a valid point. I played around with the numbers before I chose that hourly rate. For example, if I reduce the number of seconds to process each comment by half and at the same time increase the hourly rate up to $50 the overall resulting cost is still a five-digit number. For that reason I will stick with these values for the rest of this blog post. I have posted the raw data so if someone wants to review the values using different input parameters they can certainly do so.

In summary:

Total comments blocked: 90,611
Total savings: $13,212.50

Adding to the Misery

Here are some numbers from other boards or blogs where I have the same code in place. All of these numbers are based on $35 / hour and 15 seconds per spam content element, whatever it might be.

Wife’s sewing blog: blocked 625 comments
My professional blog: blocked 1589 comments
Photography blog: blocked 217 comments
Other personal blog: blocked 94 comments

Total comments blocked: 2,525
Savings: $367.50
Total running savings: $13,580.00

How about web site comments? I have comment forms on two different domains and have implemented the same protection mechanism on both. Here are the numbers:

Web site comment form: blocked 11046 spam comments
Web site comment form (different domain): 251 spam comments

Total comments blocked: 11,297
Total savings: $1,648.50
Total running savings: $15,228.50

What about phpBB2 registration spammers? If they register and I have to clean them up, that’s a substantially longer process than 15 seconds. If they register and post, then that can run up the clock too. Fortunately due to my Spammer Hammer clean-up process, it’s not too bad, so I will stick with the 15 second rule for these too. Here are the numbers:

phpBB2 board #1: blocked 13,806 spam registrations
phpBB2 board #2: blocked 4,973 spam registrations
phpbb2 board #3: blocked 1,994 spam registrations
phpbb2 board #4: blocked 10,894 spam registrations
phpbb2 board #5: blocked 1,454 spam registrations

Total registrations blocked: 33,121
Total savings: $4,830
Total running savings: $20,058.50

I think I’ve made my point and will stop here.

True Cost of Spam

Do I really have $20,000 in the bank because of the few simple checkboxes that appear on my registration and comment forms? No, unfortunately I do not. If I can figure out how to do that, I would be ready to retire soon. The point I am trying to make is that spammers cost time even when they are not successfully posting content on your board or blog. That time has a definite value associated with it, and in my case it seems to be fairly substantial.

The numbers I used to quantify the cost of spam don’t include things like potential lost revenues from advertisers, lost member traffic, or even loss of page rank because of spam content. I will be honest; if I were still processing over a hundred Akismet quarantined comments every morning for this blog, it (this blog) would have died a long time ago. What is the cost of losing an entire site or service because of spam? That’s much harder to quantify.

Conclusion

The time I spend writing code has a cost. The time I spend writing blog posts has a cost. The time I spend reading other blogs or discussion boards has a cost. But these are all costs that I “opt in” to paying because they are things I want to do. I don’t have any desire to waste time processing spammer data, so that’s a cost that I want to manage or reduce. It seems that the time spent writing the Checkbox Challenge code has had a very nice return on the investment.

As part of an experiment and a desire to capture more seed data for the upcoming relaunch of the bbProtection service I set up a phpBB2 board with no protection other than what is built in to the software. I have enabled user activation and I have activated the visual confirmation. I launched the board on August 15. Within 48 hours I had my first spam registration and my first spam post. The honey pot process has started slow but I’m getting an average of four registrations a day so far. Nine of the 17 have posted at least once (over 50% ratio). None of the posts are anything you would want your children to see; it’s really nasty stuff.

The only MODs I’ve applied to this board are a MOD to capture the IP address during the registration process (in case the bot doesn’t post I still want to know where they’ve come from) and to add the “nofollow” attribute to every link. If google finds this board I don’t want to be penalized for all of the nastiness on the other end of the outbound links.

I’ll be back in a month to post more statistics about the board. It should be interesting.

At this point I would like to mention that I was invited to join the bbProtection team a few weeks ago and I accepted a limited role. I won’t be doing any coding (at least that’s the plan). My role is more of an enthusiastic user than anything else, I guess. I have offered my input as to the relative value of some of the features being considered and suggested some others. I hope to be able to provide some value as far as the database design and tuning, as that’s where my main expertise lies.

Why mention this now? Because if you do pop in to the IRC channel as discussed on the team blog, I may be there as one of the team members that you see. I don’t go into IRC every day but if I am signed on, I will be in the channel. If you have any concerns or comments about the service I would be happy to hear them, as would any of the other team members.

See you there.

That’s where a service like AKismet (for Wordpress) or bbProtection (for bulletin boards) comes in. The bbProtection service started in 2006, and it showed a lot of promise. Like any new effort there were bumps along the way, including some concerns about privacy rights and poison pills and other issues. The folks behind bbProtection revised some of their methods made improvements based on public feedback, but ultimately (and unfortunately in my opinion) folded in 2007 due to time constraints.

Fast forward to Londonvasion 2008… After my anti-spam talk I had an interesting conversation with Mark Barnes (MarkTheDaemon). He was one of the principals behind the original effort, and he along with some of the other original team are determined to bring the service back. In fact, here is what the site looks like today.

bbprotection.net

I’ve had several conversations with Mark and Vic D’Elfant (another one of the original principals) over the past weeks about the service. I am encouraged by what I’ve learned, and look forward to the relaunch of the service. Based on their current site if you have any interest in this area and have some time to dedicate you should contact them at the email address provided and see what you can do to help.