First I added some javascript to the page (but only for that topic) that made the Reply and Quote buttons move away from the mouse. That made it impossible to click on the button, but you could still tab to the buttons and invoke the required code. Yesterday I switched the normal images for the buttons with the spacer.gif and sized it to zero by zero pixels, essentially making the button invisible. I also altered the tab index to -1 which according to a few sites I read makes the button disappear from the tab sequence.

Of course there are still several ways for folks to post in the topic. That’s sort of the point, to see how long it takes folks to figure out how to work around the challenges I have put in place. For example in the first version someone could disable javascript and the buttons would no longer move, giving them another way to click the button rather than using the tab key.

To continue the fun, I am looking for suggestions for other ways to challenge folks, and keep them from posting in that one topic. The key is there has to be some sort of loophole, as I’m not trying to completely lock folks out.

Any ideas?

At the time I didn’t have a good explanation (or even a theory) as to why this was happening. As luck would have it, I’m now getting zero dollars from Adsense for Search and am still waiting for a response from Google regarding the matter. While trying to determine what the issue could be, I have been reading the support forums for Adsense. I saw a post that suggested why my search revenues were rising.

Adsense Drivers

Content ads are driven by – naturally enough – the content on my web pages. Who is responsible for the content? Normally that would be me. In the case of a web site that includes a phpBB board, however, the content is created by anyone who participates in a topic. So any board member is creating content for my site.

But are those board members potential customers? Is the content they create suitable for advertising? In many cases the answer is no.

Board Customers Versus Board Members

But what about a board “customer” rather than a board member? I have statistics that show that about 80% of the visitors to my site are guests. They’re not providing any content at all, they’re consuming the content that is already there. They’re looking for something, whether it’s the answer to a question or something else. And how do these customers find what they’re looking for?

They search.

And if they use the Google search widget at the bottom of my page, then Google is quite aware of the key words they entered during the process. Those key words are words that the customer is interested in right now and that’s why they’re searching for them. Let me use a quick example… suppose I ran a discussion board related to a particular type of car. There might be lots of posts about performance tires on my board, so when someone reads a topic about these products it would make sense for Google to place ads about tires on the page. However, the person reading the page might not be interested in buying tires at the moment… he might just be asking a question about the performance characteristics or something else. But when someone searches the board for information about the tires, it’s assumed they have a specific interest… perhaps they’re looking for reviews from consumers before they buy tires. The search results page becomes more valuable real estate because the person is actively looking for tire information and not just reading about it for who knows what reason.

This makes those keywords more valuable than words that a board member happens to put on the page. And since they’re more valuable, they’re willing to pay more. Because they’re paying more, I am earning more. It all makes sense.

Conclusion

Despite this revelation, I still don’t have Google search active on my site. Until I can understand why all of my revenues are all of a sudden being retained for “search costs” I will leave it off. But I like the explanation and decided to share it here as a follow up post.

See you next year.

Just a few minutes ago I was on phpbb.com and saw a post in the General Discussion with the title “Is this new home page nice?” Anyone that has been around phpbb.com for a while knows that this sort of post – even in GD – is against the rules. I figured that someone might have reported it already, but there’s no indication that such an action was taken. I decided to go ahead and report the post.

When I clicked the proper icon, here’s the message I got:

This post has already been reported.

Well. If that’s the case, why not tell me?

Reporting Posts With Feedback

I prefer my method. When a post is reported, a red “alert box” is added to that specific post, detailing when and why it was reported. It does not include who did the reporting, but that information is captured as well. Here’s what that box might look like:

This box serves a couple of purposes. First, it keeps a second (and third and fourth) person from attempting to report the post when it’s already in the queue for a moderator to review. Second, it serves as an immediate feedback to the user who posted in the wrong forum (as in the case above) and helps them learn the board rules and procedures faster.

Once the post has been acted upon, the alert box changes to show the updated status.

Suppose a person reported a post for being spam, or as in the example above for being in the wrong forum. The moderator may disagree with the assessment and decide to leave the post in the original forum. If there was no indication that this process had taken place, the post might very well be reported again. And again.

Both of the alert boxes pictured above are only shown to logged in users. Since guests can’t report posts, there’s no need for them to see this information. But any logged-in user with permissions to report a post will be told before they attempt to send the report whether it’s necessary.

Icon Explanation

The first image shown above has three icons, so I thought I would explain them briefly. The yellow flag icon allows the moderator to flag the post as “in process” or “being reviewed” for now. That means a moderator has picked up this post off of the queue and is working through the process but hasn’t decided what action to take yet. The green check icon allows the moderator to close the post and enter notes about the action(s) taken. Finally, the red X icon allows the moderator to reject the report and explain why.

If a user has accumulated a certain number of rejected reports, then their permission to report additional posts is revoked. This is to prevent someone from running around and reporting every single post they see just to be a nuisance. Over time I might also review the accept / reject ratio on post reports to determine if I want to extend an invitation to a particular user to join the moderator team.

Conclusion

It’s all about communication. The phpBB3 post report process is largely hidden. Mine is more visible. Is this a better design? I think so, but I would welcome any input, either in support of or contrary to my opinion.

The Google Float is the amount AdSense publishers have accrued in earnings but have not yet been paid. You know, all those people that haven’t reached the $100 mark that triggers a payment. The figure is currently at $532,547 million. That’s over a half billion dollars. If Google killed AdSense, they would have to pay that all out.

One interesting thing about the AdSense float is that it increased last quarter by 3%. In the 2 previous quarters it decreased. I wonder if this was due to all the AdSense accounts that were disabled recently.

Tell me there’s no incentive to start figuring out ways to avoid paying out those funds.

I myself have seen one strategy… Google simply stopped paying me. That’s not the actual process, but that is the net result. I have more than one post about how the search links from Google had been starting to pay more than Adsense for Content. A couple of months ago (September) Google fixed that: the took 100% of my search earnings (yes, that’s correct, I wrote 100% as in all of it) for a “search costs” adjustment. This fee isn’t new… or rather the ability of Google to apply this fee is not new. It’s covered in several places on their site and in their terms and conditions.

What are the fees mentioned in the Terms and Conditions?
Adjustments

The problem that I have with this is after almost five years of zero adjustments I find it irritating, frustrating, and in fact downright suspicious that Google has decided that my “costs of search” now equal my entire search revenue amount.

For two months in a row.

I sent Google an email. I got a form letter back.

I have removed Google from my sites.

With phpBB3 the moderator team can lock a post to prevent further editing. But once the original content is gone it doesn’t help. So tonight I started thinking about how and where to store post revisions in order to recover from this sort of action.

Defining the Problem

Here’s what I want to be able to do:

1. Capture the prior text of any edited post and store is somewhere
2. Track who last edited a post
3. Give moderators the ability to review the post edit history and revert back to an older version
4. Provide the ability to lock a post so further editing (except by moderators) is no longer possible

This MOD is still a work in progress so I don’t have final code to share yet. But there are a few interesting wrinkles that I thought about during the design process that I thought I would share.

Locking Posts

Locking posts is one of the easier parts. I added a status field to the phpbb_posts table. For topics there is a topic_status field that contains several different status values. I don’t need anything that complex so I called my new field post_locked and made it a tinyint(1) unsigned with a default of zero. That means that every new post that gets inserted is going to default to unlocked. Of course I added the new field to the insert statement rather than rely on the database to provide the default for me.

Once the field is present in the table I have to check it. At the beginning of viewtopic.php there are a number of authorization checks that determine which buttons / icons are displayed on the post. If the post is locked I do not display the edit button; that’s simple enough. However, what if someone creates their own URL rather than clicking the button image? In that case I have code at the top of posting.php to see if the requested function is “editpost” and then check to see if the post_locked field is set to 1 instead of 0, and if so I reject the edit attempt.

At this point I have not decided just how moderators will lock / unlock posts. One easy way would be to add a lock / unlock button on each post that moderators can see and use. However, I currently envision the locking process only being used when a board member has abused their edit permissions. That means it would be more efficient to provide my moderators a way to lock a post as they are reverting the post to a prior version.

Who Edited The Post?

With a default phpBB2 installation we don’t store who edited a post. In fact no edit history is kept at all in many cases. For example if a moderator edits a post belonging to someone else, it does not trigger the edit history. If a normal user edits a post that is the last post of a topic, that does not trigger the edit history either. The post edit history is only stored when a user edits their own post after at least one reply has been made. Since we only store the fact that a user edits their own post, there is no user_id stored. The code stores the last edit date/time as well as incrementing the overall edit count. That’s it.

To address this I added a new field called last_edit_user to the phpbb_posts table. I altered the edit history so that it runs every time a post is edited instead of only when a user edits their own post. I already have a post notes feature that records who edited the post and when. But this additional step will store the last edit user (and only the last edit user) on the post itself which means I don’t have to join out to my post notes table. I made a slight adjustment to the viewtopic.php code so that the “Last edited by…” message now includes the true user name for all edits.

What Was Changed?

This was the fun part: how do I store revisions of the post? After thinking through this and considering a number of esoteric ways to store differences in the post text I threw up my hands and decided simply to copy the entire text of the post to a new row. Why? I’m using a fraction of my disk space. I can also take advantage of the fact that in phpBB2 the post table and the post text table are separate. So here’s what I did to do that.

First I added a new field to the phpbb_posts_text table called post_version that is manditory (not null). It stores an unsigned integer value. The current version of the post text is always version zero in this table. Second, I went through the base phpBB2 code and added the following to every SQL statement that joins to the phpbb_posts_text table:

AND pt.post_version = 0

There were about 20 files that needed to have this change, but in the grand scheme of things it was a low-impact update. Once I added the column to my table and updated the code everything ran perfectly. The next step is to figure out what to store in this new field.

Tracking Versions

For most of what goes on during the post processing on a phpBB board there isn’t much to change other than the extra join column. I only ever want to search the current version. I only want to display the current version on viewtopic.php. In fact, every time I reference the phpbb_posts_text table I want only the current version… unless I am a moderator that needs to review the actual post history. Why did I make the current post version zero instead of taking the maximum number? Simple. Every post has at least one version to start with, and that version will always be zero. By ensuring that the “current” version of the post text is always stored as version 0 my join logic is extremely simple.

But what does the version number mean then? In this model, I think of the version as a number representing how many “versions ago” the text was posted. Take a post with 5 version rows in the post text table. Version 0 is the currently displayed (and search indexed) version. Version 1 is one version ago. Version 2 is two versions ago, and so on up to version 4 (the original post text) which was four versions ago compared to the current text.

How are these versions created? It’s fairly simple. In the standard phpBB2 code there is a statement that checks to see if the posting process is in edit mode or new post mode. It creates either an INSERT or UPDATE statement based on the mode. All I did was add this check:

        if ( $mode == 'editpost' )
        {
                $sql =  "UPDATE " . POSTS_TEXT_TABLE . "
                        SET     post_version = post_version + 1
                        WHERE   post_id = $post_id
                        ORDER BY post_version DESC";
                if ( !$db->sql_query($sql) )
                {
                        message_die (GENERAL_ERROR, 'Error incrementing post version', '', __LINE__, __FILE__, $sql);
                }
        }

There are a couple of interesting things here. During the edit process I increment the post version for every existing record for the post by one. This means 1 becomes 2, 2 becomes 3, and so on. This generates a SQL error because when 1 becomes 2 and 2 already exists it violates the unique primary key constraint. I added an ORDER BY clause to the update statement to ensure that I start with the end of the chain instead and it fixed that problem. By starting at the end, 3 becomes 4 before 2 becomes 3 and the constraint is never violated.

This may or may not be cross-database compatible; I don’t know which databases allow an update to have an ORDER BY clause.

After the post version numbers are incremented I always do an insert because post version zero no longer exists. That means that posts are never edited… they are always inserts. This means fewer database locks which is also a good thing.

Why Not Store Version On the Post Table?

I am going to anticipate the following question because I think it’s quite logical. In fact, I reviewed this idea myself before rejecting it in favor of what I ultimately decided on. The question?

Why not store the post version on the post table and increment it during edits, rather than using the backwards zero-based logic?

This is a good question, I think. But it’s considered bad form to ever update a database key. Once it’s set it should never change. If the post_id + post_version combination key in the post table ever got out of sync with the post text table I would have a problem. By using only the post_id as the main key and the post_version as a qualifier (and only storing the post_version in one place) I will never have that problem.

Would it have saved me some work? The only benefit is that the post table itself would store an indication of how many versions there are. But wait, I have that already. When I altered the edit process to that every edit increments the post_edit_count field I essentially got that key value. Why not use it in my join instead?

I will say it again: updating a key is a bad idea. The post edit count is informative but should not be used as a key. What happens if there is a database hiccup between the update of the posts table and the insert to the posts_text table? The edit count might be updated to 5 but I only have rows 1-4 in my post text table, and because of that the join fails. In the solution I decided to go with, I will only ever have a problem if the initial post insert process fails, otherwise I will always have a post version of zero stored in my table.

What’s Next?

At this point I have created the field needed to allow moderators to lock a post and I have written the code to check this field and prevent users from editing their post once it has been locked. I have not written the code that would allow a moderator to lock / unlock the post.

I have updated the code that tracks edits and added the last edit user to the posts table. I have also updated the code so that every edit – not just the qualifying edits as determined by the standard phpBB2 code – will trigger this code.

Finally, I have altered the posts text table so that it includes a version tag, updated all of the base phpBB2 code to reference version zero, and altered the posting process so that every edit is an insert rather than an edit to the table.

At this point I have opened a discussion with my moderator team to design the interface that they will use to interact with this new information. I need to have some way to indicate to the team that edits have been performed (already in place with the edit post notes). I need to have some way for the team to open the post history and review it. I need to allow them to decide which version of the text to revert to, and I need to provide them the option to lock the post to that text once the revert process is done.

I will be sure to post an update as the discussion moves forward. If anyone has done this before and would be willing to share your own decision ideas / process I would certainly be interested in hearing from you.

That’s not something I am really happy about.

There are other instances of hosted code on my site. For example, there is a small bit of javascript that runs the Google Adsense content on the bottom of the page. So far I have not had any problems with that other than the occasional performance problem. But the AddThis code became a nuisance a few weeks ago, and today it flat-out broke my board, causing the “back” button on many versions of IE to stop working. That’s not acceptable. For this reason I have permanently removed this code from my board.

First Symptom

A few weeks ago the “newest post” link stopped working. Something was interjecting a stray URL in the middle of the page that caused the page URL to break. Instead of this:

www.myboard.com/viewtopic.php?p=123#123

I would end up with this:

www.myboard.com/viewtopic.php?p=123#

That doesn’t work, so the newest post logic failed. This was a nuisance, but nobody complained about it, so I was wondering if it was just me. I found out today it was doing this for other folks too and nobody complained.

Next Symptom

Today a new release of AddThis was released, and it broke the back button for IE6 and IE7. That spawned immediate feedback from my users. Some of them did a search and figured out what was going on, and with a quick edit of my files I permanently removed the AddThis code. I don’t plan to put it back.

Hosted Code

Frankly this could happen with any sort of hosted code. Even if you assume that the code provider is trustworthy, there is a risk involved. What sort of testing do you do with new code before you release it? How much feedback do you get from your users? What sort of regression testing plans do you have? Now, do you think that your hosted code goes through the same series of tests that you do on your own code? What if they add new features and don’t tell you about them? What if they introduce bugs, like AddThis did?

I have read several articles that suggest we should all use Google’s hosted code for their AJAX framework. I didn’t like that idea to begin with, and I like it even less now. You can be sure that from here on out I will be very careful about adding references to hosted code to my boards.

Yes, that is an actual CAPTCHA image that I was presented with. If anyone can figure out what that one is supposed to be saying, you have better eyes than I do.

These challenges are designed – in theory – to make it harder for automated processes or “bots” to use a service by requiring something like human perception or intelligence to solve a test. The full name is Completely Automated Public Turing test to tell Computers and Humans Apart. What is a Turing Test? Wikipedia says:

The Turing test is a proposal for a test of a machine’s ability to demonstrate intelligence. It proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are placed in isolated locations. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test. In order to test the machine’s intelligence rather than its ability to render words into audio, the conversation is limited to a text-only channel such as a computer keyboard and screen.

The general concept is that the test or challenge is designed to weed out computer bots from real humans. The problem is bots are often better at solving problems than humans are, and even if they aren’t, they have a lot more patience.

As a board owner, there is a fine line to walk here. I want my users to be able to register. I don’t want bots to be able to register. Anything that makes it harder for bots is also likely to make it harder for users. When the scales tip to where the inconvenience to my potential new users outweighs the bot protection then I have a problem. In my opinion, some CAPTCHA techniques tip the scale in that direction, especially some of the more complex image challenges. I’m going to save talking about image CAPTCHAs for another post and focus on alternate methods. I am going to pick three tests and try to propose how easy they are for humans to solve, and how susceptible I think they are to bots. Those methods are question/answer, picture or “kitten auth” method, and my own checkbox challenge.

Question / Answer

This technique was introduced during the phpBB2 days and is much easier to manage with phpBB3 since a board owner can set up custom registration fields. The basic premise is this: the board owner sets up a question on the registration page that requires an answer. The answer could be provided in the form of a drop-down list or other input control, or alternatively it could be an open text field that requires the user to enter the answer manually. The question can be related to the primary subject matter for the board or it could be a general knowledge question like what is 2 + 2 or what color is the sky. In any case, the question is supposed to be easily answered by a human and impossible to answer for a bot. Let’s look at some examples.

Finite Result Set

If the question is presented with a set of options, either via a drop down, radio grouping, or some other interface element, it reduces the risk that a human will fail the test. It also improves the success rate for bots. Let me present a simple example. The form below presents a question and a set of options.

As a typical human I should not have any trouble answering the question. I specifically left out “black” as a color choice, because some people might consider the sky at night and make that choice. I left out white (confusion with clouds) and some other colors for the same reason. My goal is to get the user to select “Blue” as the proper answer to this question. I would guess that 99.9% of humans would be able to pass this test.

Another advantage to this particular question is there’s no regional or subject-matter bias. No matter where you are on the planet, as long as you can read English you should be able to identify with this question and select the proper answer.

This challenge also fairs well for the visually impaired. A screen-scraper will be able to present this challenge and the user should be able to solve it with the information available. While the number of visually impaired people as a percentage of total users of the Internet is certainly quite small, it’s nice to consider their needs.

As a board owner I could provide a question that is more specific to my audience. Suppose that my board audience is made up of electrical engineers. I might present them with a series of color codes and ask them to identify the resistor rating. If my board audience is made up of fans of a particular music artist I might ask them to identify the first hit song for that artist. Knitters could get a question about yarn. Car enthusiasts could get a question about engine technologies. The popularity of this solution is partially based on the fact that the question can be as hard or easy as you want. As a result, the number of options are essentially infinite.

Bots versus Question / Answer Challenge

So far the question / answer challenge seems to do okay at allowing humans to register. How will it do as a bot preventative?

In my opinion, as it has been presented so far, it has a number of issues. The first issue is that there is a finite list of choices to make. The list has six entries: Red, Orange, Yellow, Green, Blue, and Purple. Most unsophisticated bots will pick the first option so it’s important that “Blue” (the correct answer) is not the first on the list. As I have to allow for a user to read the form incorrectly at least once, I should not block or ban the registration after the first failure. In fact I might allow two or three registration attempts before taking any action. With six answers and assuming a bot is smart enough to make different selections as it goes through the form, there is a 50% chance that a bot can “learn” or “guess” the right answer on the first series of registration attempts. Since there are only six answers (and the answer set does not change each time) there is a 100% chance that the bot will be able to register if six attempts are allowed.

What about a different interface choice?

This doesn’t solve the problem; it’s just a different interface choice (radio instead of drop-down control).

How about an open text box?

This interface choice presents an interesting dilemma. On the one hand, a bot can’t use brute force to solve this challenge. There are no options given so the answer must be determined by some other means. Problem solved, yes?

No. Believe it or not, I have read some articles that suggest bots are sophisticated enough to plug unknown questions into a search engine and get the answer that way. When I plug the question “What color is the sky” into Google, the top three results all mention the word “color” and the word “blue” in close proximity. A reasonably sophisticated bot could figure this out. If this particular technique (question / answer with open input field) were to become widely used on the Internet, I have no doubt that bots would very soon be able to handle this challenge as well as (or perhaps better than) humans.

Earlier I suggested that humans should be able to solve the drop-down challenge nearly 100% of the time, certainly with two attempts. With the open text field that percentage would almost certainly drop. Let me examine that a bit further.

Validating Input

Here are some answers that I would anticipate coming into my form if I used the open text box version of the question / answer challenge.

Q: What color is the sky?
blue
Blue
BLUE
Bleu
blu
BLU

… and so on with the variations. Hm. Do I see a problem here? When left on their own, users are going to provide a wide variety of answers that probably should be allowed but won’t be under a strict comparison to the expected answer Blue. I would expect variations in case, in spelling, and perhaps even answers with extra spaces or entire sentences like “The sky is blue.” Once the input becomes open for anything, then anything is what I expect to get. How can I certify these answers (all of which are reasonably correct) and allow the user to register? Ironically if a bot is able to get the answer correct, they will most certainly provide the expected spelling of “blue” rather than one of the variations shown above.

Fortunately there is a simple function that I can use to help solve most of these challenges. That function is called soundex() and I will detail it next.

Introducing The soundex() Function

Whether I see the word “blue” or “blu” or even “bleu” the sound of the word is the same. That’s what the soundex() function does; it returns a code that is supposed to designate the sound aspects of the word rather than the literal word. First I will check the soundex() result for the required answer:

mysql> select soundex('blue');
+-----------------+
| soundex('blue') |
+-----------------+
| B400 |
+-----------------+

Next I will check the results for some of the variations shown.

mysql> select soundex('bleu');
+-----------------+
| soundex('bleu') |
+-----------------+
| B400 |
+-----------------+

mysql> select soundex('blueu');
+------------------+
| soundex('blueu') |
+------------------+
| B400 |
+------------------+

mysql> select soundex('blu e');
+------------------+
| soundex('blu e') |
+------------------+
| B400 |
+------------------+

mysql> select soundex('black');
+------------------+
| soundex('black') |
+------------------+
| B420 |
+------------------+

Notice in every case except for the obviously wrong answer “black” I get the code B400. I won’t go into details of how this code is derived (there is a Wiki link at the end of the post if you want those details). I will make the observation that all of the spellings – both correct and “close enough” to correct – return the same code.

Let me review how I got to this point. A question / answer challenge with a finite list of possible answers is susceptible to brute-force solving by bots or humans. The same challenge with an open text box for the answer is much harder for bots to solve, but it also reduces the success rate for humans due to input variations. I am proposing that the soundex() function could be used to reduce the number of humans rejected because of minor spelling variations, and I do believe it would work.

It also helps the bots.

If I am expecting a single word answer and the user enters “The sky is blue” instead, I still have options. I can programmatically split up the phrase into component words and then apply the soundex() function to each one. As long as the expect word “blue” is in the phrase, I can decide to let the registration attempt succeed.

Other Styles of Questions

I have seen people propose that simple math problems are a good question. There is only one answer, right? Well, it depends.

Q: What is 2 + 2?
4
four
for
22

All of these are potential answers. And this doesn’t help against bots; try plugging 2+2= into the Google search form and see what you get.

I have seen people suggest that the question be embedded within an image like this:

This doesn’t really help either. Bots have already demonstrated a high degree of success against CAPTCHA images so putting our question into an image rather than text doesn’t really buy much. It also reduces or eliminates the ability of a visually impaired user to solve the challenge.

Summary

There are several benefits and issues with the question / answer style of CAPTCHA.

Advantages:

1. With a finite list it is very easy for the user to interact with
2. With a finite list it is very easy to validate
3. The question can be tailored to the board audience
4. The question and related answers can be maintained via a simple administrative page
5. The technique does not penalize visually impared users

Disadvantages:

1. With a finite list this technique is suceptable to brute-force attacks
2. Sophisticated bots might use search engines to solve the answer
3. Use of a text field instead of a list control provides more protection from bots but requires more complex code and impacts the user experience

The number of advantages does outweigh the disadvantages. There are quite a few fans of this technique. There are several MODs for phpBB2 that provide this feature, and phpBB3 essentially has it built-in with the custom registration fields option. I consider this to be a preferable option to the image CAPTCHA techniques that are much more prevalent today.

Next time I want to talk about the “kitten auth” technique. I hope to have that post ready soon but have been fairly busy in real life lately so please be patient if it takes a bit longer.

Related Links

• Wiki on the soundex() function

What Happened Yesterday?

One of the frequent requests that I used to see on phpbb.com was something like this:

How many visitors came to my board yesterday?

The problem I have with questions like this is that your “yesterday” is not the same as mine, unless you happen to live in the central time zone in the United States. When I wrote a MOD to do this for a client, I convinced them that rather than showing what happened “yesterday” it would be better to show what happened in the last 24 hours.

The user_lastvisit field shows the date/time that a user last logged in. This field is used to track new topics during a user session. It’s also used to drive the difference between “new” and “unread” personal messages. (A “new” message arrived since the last session. An “unread” message is one that hasn’t been read yet but arrived before the current session started.) I have altered my memberlist.php code to show when the user last visited as well.

Like most date fields in phpbb, this field is stored as int(11) rather than as a date/time field. (Other examples are the user’s registration date, the post time, new topic time. … the list goes on from there.) The content of the field is a very large integer value and is officially known as a unix timestamp.

Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.

The standard for storing date/time fields in unix timestamp is to use a signed integer rather than unsigned. This allows a developer to store negative numbers to reflect dates prior to 1970. It also has its own Y2K issue as the int(11) field will overflow in 2038. But let me get back on track for this blog post.

SQL Code for Last 24 Hours

Because of the way the user last visit time is stored, I can easily get a list of people that have visited my board in the last 24 hours with this SQL code:

select  user_id
,       username
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - 86400)
order by user_lastvisit desc

The MySQL function unix_timestamp() returns the current date and time in a unix timestamp format so I don’t have to convert anything. Since the unix timestamp is a number of seconds, and since one day has 86400 seconds, by subtracting 86400 from the current time I get the matching time from 24 hours ago. Easy stuff.

If I wanted to truly get a list of people that signed in “yesterday” then the first thing I have to do is define what yesterday means. Time zones could get involved. It could get messy. I much prefer the “last 24 hours” definition because it’s the same for everybody everywhere.

What About More Than One Day?

Sometimes I want to calculate more than one day. Instead of memorizing multiples of 86400 I simply multiply by the number of days. So if I want to count how many people have logged in for the past 7 days (as defined by 24-hour periods rather than “days”) I would do this:

select  count(user_id)
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - ( 86400 * 7 ) )

This is easy enough to do, and the code becomes “self-documenting” in a manner of speaking. I know that there are 86400 seconds in a day, and if I multiply by 7 I get a week. This is much easier to read and understand than using the number 604800.

Measuring Board Activity

About two years ago I told folks I was eagerly looking forward to the first week where my board averaged 86400 page views daily. Now that I have explained what the number is, that statement makes more sense. I was looking for the first week that I averaged a page view every second for an entire week. That happened over a year ago, and in fact my board averages over 100K page views daily at this point.

Now I am looking forward to the first week that I average 172800 page views a day. Hmm, I wonder why that is?

Related Links

• Wiki on Unix Timestamps
• Wiki on UTC
• Unix Millenium Bug