with one of several holiday logos:

And of course this:

During the American Thanksgiving week we used this:

Over at the phpBB2 Refugees site we’ve swapped our normal refugee tent for one with a more holiday feel for the last several weeks:

Right now this is a manual process, which means I have to remember to edit the php code to invoke the new logo (or alternately use ftp to upload a replacement logo with the same name over the current one). Manual process? for Dave?

I’m setting up a database table with dates and alternate logo names, and will set up a cron job that every night at midnight checks to see if holiday logos are needed for the following day or not. On BOB we rotate among several logos, so there could be more than one logo at any given time. The logo information is then written to a cache file where it will be used for the following 24 hours.

Easy as pie.

Holiday pie.

I used the same function to update my canned messages MOD as well.

Oh, and I updated the color picker just a bit. The original color picker used values of 00, 40, 80, and two more to create an array of colors that was 5×5x5. I changed it to the old “Netscape Safe” color palette and used 00, 33, 66, 99, CC, and FF and created a 6×6x6 grid instead.

I have the new posting form activated on two different boards for the moment. As long as I don’t find too many issues I will write it up as a MOD.

I think I’ve managed to come up with something that definitely helps solve the first two scenarios and as a bonus helps the spammer problem as well. I call this my “Cross Post / Double Post” MOD, and it’s being tested on my beta board now.

The MOD design has so far turned out to be fairly simple. I tie into the flood control process and retrieve the post text for the last three posts by the user. From there I take the current post text and compare it to the prior posts. The first check is a straight equality check, meaning I check for the exact same post text. This will catch the “copy/paste” folks with very little overhead. If the post text is not identical, then next I use a function called similar_text(). (similar text reference at php.net) This function takes three arguments. The first two are the two strings to compare, and the third is a variable to store the results of the comparison, which is a number from 0 to 100. The result code should essentially be treated as a percentage. If the two posts are 95% similar then I check to see if the original post already in the database is in the same forum as the new post being attempted. If the forums are the same, then a “Double post” exception is triggered. If the forums are different, then a “Cross post” exception is triggered instead.

The number of posts (3) and percentage of similarity (95) are both controlled via the board configuration screen, so it’s quite flexible. Setting the percentage threshold to zero (0) is the same as turning the comparison process off.

This MOD is being tested on my “beta release” board right now. The first version of the MOD did not use the similar_text() function mentioned above. I attempted to use the soundex() function instead. However it seemed that the soundex() function did not look at enough text, so posts that were clearly different were still being reported as being the same. Switching functions solved that issue.

I’m now wondering if I need to deal with setting different threshold values for different forums. I hate to do that, as it drastically increases the complexity of the code. But for example there are many forum “games” that people play in an “off topic” type of forum. Some of those games look very repetitive, and would potentially trigger the CP/DP exception handling. Then again, the current logic looks across all forums, so as long as the person is active in more areas than just the off-topic games area it might be okay. I don’t want this feature to get in the way of normal use, but I do want to help out the moderator team by capturing / rejecting double post and cross post events.

Stay tuned for details as we start user testing this week.

1. Full-Text Search
   I created a full-text index on the post subject and text over a year ago to see if maintaining that index would cause any performance issues. I’m happy to say that I have not seen any challenges from inserts / updates with this index in place. I’m going to be altering the search screen to allow the full syntax offered by MySQL on this type of index and hope to release that in a few months. Some of the challenges I have not yet decided how to solve are things like limiting forums – either by security or user preference – and other criteria that can be entered on the standard search screen.
2. Capture Post Revisions
   I’ve also added some code to capture post revisions. We’ve had a couple of folks that come back to our board and edit their post, removing all of the text and leaving only something like “…” instead. This destroys the continuity of the topic, and as a result we’re going to now track post revisions by capturing the post text history. If needed a moderator will be able to review and then restore a prior post, and ultimately lock that post from further editing. As with the full text search I have done fairly extensive testing on how this is implemented in order to ensure that performance does not suffer, and I’ll have a few blog posts about that process. This MOD is completed and I expect to roll it out onto the main board in a few weeks. (FWIW, I first talked about this post several years ago, and am just now finally getting it completed.)
3. Moderator Posts
   I’ve added a new field to the post table that allows a moderator to designate whether it’s a moderator post or a user post. For example, moderators can certainly participate in a normal board conversation as a regular person. But they may also add posts in their role as a moderator. This new feature will format those posts differently so they stand out, will automatically remove the “personal” aspects of a post such as signatures, and does not increment a moderator post count for this type of post. It is intended to be a way for moderators to be able to separate out their moderator posts from their board participation posts. This MOD is also completed and expected to be released shortly.
4. Including External Content
   I’ve added some cron jobs that parse RSS feeds from several blogs owned by board members. Their blog posts are automatically set up as part of their signature (as “Latest Blog Posts”) and updated once an hour. For bloggers that our community wants to recognize, this is a great way for them to get additional exposure without having to manually update their signature every time they publish a new blog post. This part of the MOD is already in use on our board. Only board admins can currently enter blogger information, as we want to go through a review process and certify blogs rather than allowing just anybody to link to an external site. This was done by altering the administrator user edit form and leaving the regular user profile form alone.

   As an extension to this, I’m also pulling in the content from the blog post and storing that in a hidden forum. As the blog posts are added to the forum they are obviously added to the full-text index because they’re part of the same table. I am also adding these posts to the standard phpBB2 search tables at the same time. That way if someone searches for term “X” and that’s found in an external blog post, they’ll see a link in their search results. The blog address is stored on the topic table and a different icon is used to show the user that they’re leaving our board and heading to an external site. I have all of the main work done; the last requirement is altering search.php so that it offers the ability to include / exclude external content and then react to that setting accordingly. I hope to get this completed in the next few weeks.

5. Social Media Profile Links
   I’ve added Facebook, Twitter, and LinkedIn fields to user profiles. These are displayed along with the other profile links, using smaller 18×18 pixel logos. I’m planning on going back and redoing the other profile links to use the same form factor but that part hasn’t been done yet. Here are the images I’ve made, using logos or other material provided by each service provider.

One thing that many of these MODs have in common is my concern for performance. We’re over 750K posts now, and still running extremely well on a server that is hosting several dozen sites, although none of them as active as our big board. Every time I touch the code performance is a primary goal. Another MOD that I’ve been planning is to port the phpBB3 posting form back to phpBB2 since it does a better job of supporting modern browsers as well as proving some additional formatting features. I haven’t even started on that yet, but I think it would be good. Now that I’ve personally switched to Chrome as my standard browser I’m noticing some interesting quirks.

So that’s what I’ve been up to for the past few months.

First I added some javascript to the page (but only for that topic) that made the Reply and Quote buttons move away from the mouse. That made it impossible to click on the button, but you could still tab to the buttons and invoke the required code. Yesterday I switched the normal images for the buttons with the spacer.gif and sized it to zero by zero pixels, essentially making the button invisible. I also altered the tab index to -1 which according to a few sites I read makes the button disappear from the tab sequence.

Of course there are still several ways for folks to post in the topic. That’s sort of the point, to see how long it takes folks to figure out how to work around the challenges I have put in place. For example in the first version someone could disable javascript and the buttons would no longer move, giving them another way to click the button rather than using the tab key.

To continue the fun, I am looking for suggestions for other ways to challenge folks, and keep them from posting in that one topic. The key is there has to be some sort of loophole, as I’m not trying to completely lock folks out.

Any ideas?

Just a few minutes ago I was on phpbb.com and saw a post in the General Discussion with the title “Is this new home page nice?” Anyone that has been around phpbb.com for a while knows that this sort of post – even in GD – is against the rules. I figured that someone might have reported it already, but there’s no indication that such an action was taken. I decided to go ahead and report the post.

When I clicked the proper icon, here’s the message I got:

This post has already been reported.

Well. If that’s the case, why not tell me?

Reporting Posts With Feedback

I prefer my method. When a post is reported, a red “alert box” is added to that specific post, detailing when and why it was reported. It does not include who did the reporting, but that information is captured as well. Here’s what that box might look like:

This box serves a couple of purposes. First, it keeps a second (and third and fourth) person from attempting to report the post when it’s already in the queue for a moderator to review. Second, it serves as an immediate feedback to the user who posted in the wrong forum (as in the case above) and helps them learn the board rules and procedures faster.

Once the post has been acted upon, the alert box changes to show the updated status.

Suppose a person reported a post for being spam, or as in the example above for being in the wrong forum. The moderator may disagree with the assessment and decide to leave the post in the original forum. If there was no indication that this process had taken place, the post might very well be reported again. And again.

Both of the alert boxes pictured above are only shown to logged in users. Since guests can’t report posts, there’s no need for them to see this information. But any logged-in user with permissions to report a post will be told before they attempt to send the report whether it’s necessary.

Icon Explanation

The first image shown above has three icons, so I thought I would explain them briefly. The yellow flag icon allows the moderator to flag the post as “in process” or “being reviewed” for now. That means a moderator has picked up this post off of the queue and is working through the process but hasn’t decided what action to take yet. The green check icon allows the moderator to close the post and enter notes about the action(s) taken. Finally, the red X icon allows the moderator to reject the report and explain why.

If a user has accumulated a certain number of rejected reports, then their permission to report additional posts is revoked. This is to prevent someone from running around and reporting every single post they see just to be a nuisance. Over time I might also review the accept / reject ratio on post reports to determine if I want to extend an invitation to a particular user to join the moderator team.

Conclusion

It’s all about communication. The phpBB3 post report process is largely hidden. Mine is more visible. Is this a better design? I think so, but I would welcome any input, either in support of or contrary to my opinion.

With phpBB3 the moderator team can lock a post to prevent further editing. But once the original content is gone it doesn’t help. So tonight I started thinking about how and where to store post revisions in order to recover from this sort of action.

Defining the Problem

Here’s what I want to be able to do:

1. Capture the prior text of any edited post and store is somewhere
2. Track who last edited a post
3. Give moderators the ability to review the post edit history and revert back to an older version
4. Provide the ability to lock a post so further editing (except by moderators) is no longer possible

This MOD is still a work in progress so I don’t have final code to share yet. But there are a few interesting wrinkles that I thought about during the design process that I thought I would share.

Locking Posts

Locking posts is one of the easier parts. I added a status field to the phpbb_posts table. For topics there is a topic_status field that contains several different status values. I don’t need anything that complex so I called my new field post_locked and made it a tinyint(1) unsigned with a default of zero. That means that every new post that gets inserted is going to default to unlocked. Of course I added the new field to the insert statement rather than rely on the database to provide the default for me.

Once the field is present in the table I have to check it. At the beginning of viewtopic.php there are a number of authorization checks that determine which buttons / icons are displayed on the post. If the post is locked I do not display the edit button; that’s simple enough. However, what if someone creates their own URL rather than clicking the button image? In that case I have code at the top of posting.php to see if the requested function is “editpost” and then check to see if the post_locked field is set to 1 instead of 0, and if so I reject the edit attempt.

At this point I have not decided just how moderators will lock / unlock posts. One easy way would be to add a lock / unlock button on each post that moderators can see and use. However, I currently envision the locking process only being used when a board member has abused their edit permissions. That means it would be more efficient to provide my moderators a way to lock a post as they are reverting the post to a prior version.

Who Edited The Post?

With a default phpBB2 installation we don’t store who edited a post. In fact no edit history is kept at all in many cases. For example if a moderator edits a post belonging to someone else, it does not trigger the edit history. If a normal user edits a post that is the last post of a topic, that does not trigger the edit history either. The post edit history is only stored when a user edits their own post after at least one reply has been made. Since we only store the fact that a user edits their own post, there is no user_id stored. The code stores the last edit date/time as well as incrementing the overall edit count. That’s it.

To address this I added a new field called last_edit_user to the phpbb_posts table. I altered the edit history so that it runs every time a post is edited instead of only when a user edits their own post. I already have a post notes feature that records who edited the post and when. But this additional step will store the last edit user (and only the last edit user) on the post itself which means I don’t have to join out to my post notes table. I made a slight adjustment to the viewtopic.php code so that the “Last edited by…” message now includes the true user name for all edits.

What Was Changed?

This was the fun part: how do I store revisions of the post? After thinking through this and considering a number of esoteric ways to store differences in the post text I threw up my hands and decided simply to copy the entire text of the post to a new row. Why? I’m using a fraction of my disk space. I can also take advantage of the fact that in phpBB2 the post table and the post text table are separate. So here’s what I did to do that.

First I added a new field to the phpbb_posts_text table called post_version that is manditory (not null). It stores an unsigned integer value. The current version of the post text is always version zero in this table. Second, I went through the base phpBB2 code and added the following to every SQL statement that joins to the phpbb_posts_text table:

AND pt.post_version = 0

There were about 20 files that needed to have this change, but in the grand scheme of things it was a low-impact update. Once I added the column to my table and updated the code everything ran perfectly. The next step is to figure out what to store in this new field.

Tracking Versions

For most of what goes on during the post processing on a phpBB board there isn’t much to change other than the extra join column. I only ever want to search the current version. I only want to display the current version on viewtopic.php. In fact, every time I reference the phpbb_posts_text table I want only the current version… unless I am a moderator that needs to review the actual post history. Why did I make the current post version zero instead of taking the maximum number? Simple. Every post has at least one version to start with, and that version will always be zero. By ensuring that the “current” version of the post text is always stored as version 0 my join logic is extremely simple.

But what does the version number mean then? In this model, I think of the version as a number representing how many “versions ago” the text was posted. Take a post with 5 version rows in the post text table. Version 0 is the currently displayed (and search indexed) version. Version 1 is one version ago. Version 2 is two versions ago, and so on up to version 4 (the original post text) which was four versions ago compared to the current text.

How are these versions created? It’s fairly simple. In the standard phpBB2 code there is a statement that checks to see if the posting process is in edit mode or new post mode. It creates either an INSERT or UPDATE statement based on the mode. All I did was add this check:

        if ( $mode == 'editpost' )
        {
                $sql =  "UPDATE " . POSTS_TEXT_TABLE . "
                        SET     post_version = post_version + 1
                        WHERE   post_id = $post_id
                        ORDER BY post_version DESC";
                if ( !$db->sql_query($sql) )
                {
                        message_die (GENERAL_ERROR, 'Error incrementing post version', '', __LINE__, __FILE__, $sql);
                }
        }

There are a couple of interesting things here. During the edit process I increment the post version for every existing record for the post by one. This means 1 becomes 2, 2 becomes 3, and so on. This generates a SQL error because when 1 becomes 2 and 2 already exists it violates the unique primary key constraint. I added an ORDER BY clause to the update statement to ensure that I start with the end of the chain instead and it fixed that problem. By starting at the end, 3 becomes 4 before 2 becomes 3 and the constraint is never violated.

This may or may not be cross-database compatible; I don’t know which databases allow an update to have an ORDER BY clause.

After the post version numbers are incremented I always do an insert because post version zero no longer exists. That means that posts are never edited… they are always inserts. This means fewer database locks which is also a good thing.

Why Not Store Version On the Post Table?

I am going to anticipate the following question because I think it’s quite logical. In fact, I reviewed this idea myself before rejecting it in favor of what I ultimately decided on. The question?

Why not store the post version on the post table and increment it during edits, rather than using the backwards zero-based logic?

This is a good question, I think. But it’s considered bad form to ever update a database key. Once it’s set it should never change. If the post_id + post_version combination key in the post table ever got out of sync with the post text table I would have a problem. By using only the post_id as the main key and the post_version as a qualifier (and only storing the post_version in one place) I will never have that problem.

Would it have saved me some work? The only benefit is that the post table itself would store an indication of how many versions there are. But wait, I have that already. When I altered the edit process to that every edit increments the post_edit_count field I essentially got that key value. Why not use it in my join instead?

I will say it again: updating a key is a bad idea. The post edit count is informative but should not be used as a key. What happens if there is a database hiccup between the update of the posts table and the insert to the posts_text table? The edit count might be updated to 5 but I only have rows 1-4 in my post text table, and because of that the join fails. In the solution I decided to go with, I will only ever have a problem if the initial post insert process fails, otherwise I will always have a post version of zero stored in my table.

What’s Next?

At this point I have created the field needed to allow moderators to lock a post and I have written the code to check this field and prevent users from editing their post once it has been locked. I have not written the code that would allow a moderator to lock / unlock the post.

I have updated the code that tracks edits and added the last edit user to the posts table. I have also updated the code so that every edit – not just the qualifying edits as determined by the standard phpBB2 code – will trigger this code.

Finally, I have altered the posts text table so that it includes a version tag, updated all of the base phpBB2 code to reference version zero, and altered the posting process so that every edit is an insert rather than an edit to the table.

At this point I have opened a discussion with my moderator team to design the interface that they will use to interact with this new information. I need to have some way to indicate to the team that edits have been performed (already in place with the edit post notes). I need to have some way for the team to open the post history and review it. I need to allow them to decide which version of the text to revert to, and I need to provide them the option to lock the post to that text once the revert process is done.

I will be sure to post an update as the discussion moves forward. If anyone has done this before and would be willing to share your own decision ideas / process I would certainly be interested in hearing from you.

What Happened Yesterday?

One of the frequent requests that I used to see on phpbb.com was something like this:

How many visitors came to my board yesterday?

The problem I have with questions like this is that your “yesterday” is not the same as mine, unless you happen to live in the central time zone in the United States. When I wrote a MOD to do this for a client, I convinced them that rather than showing what happened “yesterday” it would be better to show what happened in the last 24 hours.

The user_lastvisit field shows the date/time that a user last logged in. This field is used to track new topics during a user session. It’s also used to drive the difference between “new” and “unread” personal messages. (A “new” message arrived since the last session. An “unread” message is one that hasn’t been read yet but arrived before the current session started.) I have altered my memberlist.php code to show when the user last visited as well.

Like most date fields in phpbb, this field is stored as int(11) rather than as a date/time field. (Other examples are the user’s registration date, the post time, new topic time. … the list goes on from there.) The content of the field is a very large integer value and is officially known as a unix timestamp.

Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.

The standard for storing date/time fields in unix timestamp is to use a signed integer rather than unsigned. This allows a developer to store negative numbers to reflect dates prior to 1970. It also has its own Y2K issue as the int(11) field will overflow in 2038. But let me get back on track for this blog post.

SQL Code for Last 24 Hours

Because of the way the user last visit time is stored, I can easily get a list of people that have visited my board in the last 24 hours with this SQL code:

select  user_id
,       username
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - 86400)
order by user_lastvisit desc

The MySQL function unix_timestamp() returns the current date and time in a unix timestamp format so I don’t have to convert anything. Since the unix timestamp is a number of seconds, and since one day has 86400 seconds, by subtracting 86400 from the current time I get the matching time from 24 hours ago. Easy stuff.

If I wanted to truly get a list of people that signed in “yesterday” then the first thing I have to do is define what yesterday means. Time zones could get involved. It could get messy. I much prefer the “last 24 hours” definition because it’s the same for everybody everywhere.

What About More Than One Day?

Sometimes I want to calculate more than one day. Instead of memorizing multiples of 86400 I simply multiply by the number of days. So if I want to count how many people have logged in for the past 7 days (as defined by 24-hour periods rather than “days”) I would do this:

select  count(user_id)
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - ( 86400 * 7 ) )

This is easy enough to do, and the code becomes “self-documenting” in a manner of speaking. I know that there are 86400 seconds in a day, and if I multiply by 7 I get a week. This is much easier to read and understand than using the number 604800.

Measuring Board Activity

About two years ago I told folks I was eagerly looking forward to the first week where my board averaged 86400 page views daily. Now that I have explained what the number is, that statement makes more sense. I was looking for the first week that I averaged a page view every second for an entire week. That happened over a year ago, and in fact my board averages over 100K page views daily at this point.

Now I am looking forward to the first week that I average 172800 page views a day. Hmm, I wonder why that is?

Related Links

• Wiki on Unix Timestamps
• Wiki on UTC
• Unix Millenium Bug

I discovered that one solution was clearly better than the other, but only if the proper index was created.

Disclaimer: I tested on phpBB2. The index that I created does not exist in a standard phpBB2, nor does it exist in a standard phpBB3 install, so I suspect this post applies to both.

Defining the Problem

Here is a partial quote of the original question in the phpBB3 MOD forum:

I want to select 5 random memebers from last 30 active users.

Simple enough, yes? First, get the last 30 members that have visited the board, then randomly select 5 of those. This could easily be done procedurally via php code, but it can also be done directly in the database with the correct SQL code.

Order By Random

The first suggestion given was this:

$start = 0;
$number = 30;
$sql = 'SELECT *
    FROM ' . USERS_TABLE . '
    ORDER BY user_lastvisit  DESC, RAND()'  ;
    $result = $db->sql_query_limit($sql, $number, $start);

With apologies to evil<3 who posted this there are a couple of things that I suggested changing. First is to avoid using the * to select every column in the table unless it’s absolutely needed. In this case, I made the assumption that the original poster was looking for a way to display a random five “recent visitors” somewhere on a page on the site. To do this doesn’t require every single bit of information about the user, just certain columns. If you select * then the entire row is returned. There are 73 columns in a standard phpBB3 users table, several of them varchar(255), and in my test installation all of the fields are mandatory. That means every single column has a value, even if it is just a space or other placeholder value. By setting up a specific list of columns to request in the query the amount of I/O is reduced. Less I/O should mean if all other things are equal the query will run faster because there are fewer bits and bytes to move around.

The other issue with the query as provided is that it’s not very efficient. It has two order by columns, one of which cannot possibly be indexed. (You can’t index something that doesn’t exist until the runtime of the query, so the rand() function result is impossible to tune.) Here is an abbreviated display for the explain plan for this query:

+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| select_type | type | possible_keys | key  | key_len | ref  | rows  | Extra                           |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| SIMPLE      | ALL  | NULL          | NULL | NULL    | NULL | 43367 | Using temporary; Using filesort |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+

This shows that no indexes will be used for this query at all, which is not good. I ran this query five times in a row on my large user table and got execution times of 0.12, 0.12, 0.12, 0.13, and 0.12 seconds.

Select Last 30 Then Random 5

As you can see from the explain data above, I have 43,367 rows in my users table right now, which is fairly large. Instead of scanning the entire table, it would be much more efficient to get the last 30 visitors in one pass and then pick five members randomly from that list. I suggested this SQL to do that:

SELECT  u.user_id
,       u.username
FROM    phpbb_users u
,       (SELECT user_id
         FROM   phpbb_users v
         ORDER BY user_lastvisit desc limit 30) v
WHERE   u.user_id = v.user_id
ORDER BY rand() limit 5;

This is an interesting technique called “inline tables” as I am creating a new table on the fly by writing SQL code inside the FROM clause. Every database I have worked with supports this technique, so it should be portable. (I do not count Microsoft Access as a real database. ) What this SQL code will do is run the inline table to return a list of 30 users, then join that virtual table to the real table by user_id (which is a unique key) and randomly select five users from the joined result set.

Is it more efficient?

I ran this query five times (as I did with the other one) and got run times of 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. It seems that it’s not really that much more effective, so is there really a clear winner?

Index Key Columns

I ran this query:

show indexes from phpbb_users

Side Note: I run my SQL directly on the database using the MySQL command line, rather than phpMyAdmin. If you use the GUI interface, then you can check for keys by looking at the appropriate screen instead of doing as I documented here.

The results of the query did not show an index on user_lastvisit which is crucial to this solution. Here is the explain plan for my query without the index:

+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key     | ref       | rows  | Extra                           |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL    | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY | v.user_id |     1 |                                 |
| DERIVED     | ALL    | NULL          | NULL    | NULL      | 43367 | Using filesort                  |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+

Notice that is also scans all 43,367 user rows. That’s okay. What isn’t okay is that it does so without the benefit of an index and it also has to do some additional work since more than one table is involved. It would seem that the first query should be more efficient since it only has one explain step and the second one has three.

However, the magic of a database indexing can fix this. The driver for this entire question is the user_lastvisit column. After creating an index on this field (which is not indexed by default in either phpBB2 or phpBB3) here is the new explain plan.

+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key            | ref       | rows  | Extra                           |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL           | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY        | v.user_id |     1 |                                 |
| DERIVED     | index  | NULL          | user_lastvisit | NULL      | 43367 |                                 |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+

Yup, still have to look at (or so the database optimizer thinks) all 43,367 users. But this time we do so with the benefit of an index. What is the impact?

The query without an index, remember, ran in 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. After creating the index I ran the same query five times and got 0.00 seconds of execution time on every trial.

Does the index help the first query? Interestingly enough, it does not. The explain plan is identical with or without the index, and the query execution times do not improve either.

However, it gets worse. It doesn’t perform as required. My query gets the last 30 users that have logged in and then randomly selects five of those. The ORDER BY clause happens after all of the select and join process is complete, so I am ordering by RAND() on at most 30 rows. The other suggestion will pick the same five users nearly every single time. Why? The secondary sort column (the “random factor”) will only come into play if two users have exactly the same last visit time (down to the second). When you have two columns in the ORDER BY clause, the first column is the primary sort and every row returned will first be sorted by that column. If there are ties in the first column then and only then will the second column be sorted.

So the first solution suggested is the worst of both worlds: not only is it slower, it is also incorrect.

Conclusion

There may be other solutions to this. I don’t mean to present this post as the ultimate answer to this question. What I hoped to accomplish with this post was to show how two solutions that look the same are not always equivalent. Subtle differences can have a huge impact on functionality.

The second lesson is that if you’re going to be asking the same question from your database over and over you should carefully consider indexing the columns used in the WHERE or ORDER BY clauses. I did some work for someone a while back (their board is one of the top ten phpBB boards on the “big boards” site). They wanted to display the top ten posters on their index. The code as written was taking over ten seconds just to run the query and then the php code / template process still had to complete. I rewrote the code and added an index on the user_posts column and the code ran in less than a hundredth of a second.

On the other hand, too many indexes can also be a problem, so don’t go out and create an index for every single column in your database. Just the ones that truly matter. In this case, it makes a substantial difference.

Implementing the Bump Warning MOD

The MOD was fairly simple to implement. It does require an extra query during the posting process. This query is used to find out if the current poster was also the last person to post in the selected topic. If so, it checks the post time to see if 24 hours have expired. If not, then it sets a value for a template switch which turns on the red warning label on the posting screen. Here’s the code from posting.php.

// BEGIN Bump Warning 1.0.0 (www.phpBBDoctor.com)
if ($mode == 'reply')
{
        $sql = 'SELECT  p.post_id
                ,       p.poster_id
                ,       p.post_time
                FROM    ' . POSTS_TABLE . ' p
                ,       ' . TOPICS_TABLE . ' t
                WHERE   t.topic_id = ' . $topic_id . '
                AND     p.post_id = t.topic_last_post_id+0';

        if (!($result = $db->sql_query($sql)))
        {
                message_die(GENERAL_ERROR, 'Invalid cursor getting last poster', '', '', '', $sql);
        }
        $check = $db->sql_fetchrow($result);
        $last_post_id = $check['post_id'];
        $db->sql_freeresult($result);

        if ( ($check['poster_id'] == $userdata['user_id']) && ($check['post_time'] > time() - (86400)) )
        {
                $template->assign_block_vars('switch_bump_warning', array(
                        'U_EDIT_PRIOR_POST' => append_sid("posting.$phpEx?mode=editpost&p=$last_post_id")
                        ));
        }
}
// END Bump Warning 1.0.0 (www.phpBBDoctor.com)

Then of course in the template file I have the switch defined.

<!-- BEGIN switch_bump_warning -->
<tr>
<td class="favrow_red" width="22%" height="35"><span class="gen"><strong>Bump Warning</strong></span></td>
<td class="favrow_red" width="78%"><span class="gen">It appears that you were the last person to post in this topic and 24 hours has not yet elapsed. Please do not reply to your post unless you have new information to add. You may also <a href="{switch_bump_warning.U_EDIT_PRIOR_POST}" class="gen">edit your prior post</a> since nobody has replied yet. Thanks.</td>
</tr>
<!-- END switch_bump_warning -->

This is all well and good, but it seems that some people can be blind. Somehow the red banner across the top of the screen isn’t enough to let them know they’re doing something that I don’t want them to do. I even give them a link to edit the prior post instead of adding a new one…

Since I have asked nicely and they’re still bumping posts, what do I do next?

Duplicate Post Prevention

There are two kinds of post bumps that I typically see. The first kind is what I have described above where the person is simply ignoring my request to not do what he or she is about to do. The second kind is also common and is not really something I can blame a user for. At times the Internet gets slow, and the posting process times out. It may also be that there are several hundred people watching a topic and it takes time to send those emails. (Something else I want to modify; I want to set up a job-queue to handle this process so the poster doesn’t pay a penalty for that during their posting process.) The bottom line is that sometimes a post times out, so the user resubmits. And resubmits. And when they’re done, there are two (or three or more) posts in a row that contain the same text.

So tonight I am starting to test a new feature. I am supplementing my bump “warning” with a bump “rejection” as well. The bump rejection does not prevent a legitimate post bump, but it will catch users that enter the same post twice due to the time-out issue I described above. As an added bonus, this new code will also detect a cross-posted topic and prevent that as well.

To make this change I opened includes/functions_post.php and added code to the Flood Control section. As this is very preliminary I am not using any language strings. That being said, here’s the code.

// Cross-post capture
$sql = 'select	post_text
	,	p.post_id
	,	p.topic_id
	from	' . POSTS_TEXT_TABLE . ' pt
	,	' . POSTS_TABLE	. ' p
	where	p.poster_id = '	. $userdata['user_id'] . '
	AND	p.post_id = pt.post_id
	ORDER BY p.post_id desc	limit 3';

if (!($result =	$db->sql_query($sql) ) )
{
	message_die(GENERAL_ERROR, 'Invalid cursor while checking for cross posts');
}

while ($cp_row = $db->sql_fetchrow($result))
{
	if (soundex($post_message) == soundex($cp_row['post_text']))
	{
		if ($topic_id == $cp_row['topic_id'])
		{
			message_die(GENERAL_MESSAGE, 'You are attempting to post the same item twice. It is possible that your browser timed-out on your prior post submission.	If you think this is an	error, please click the	"back" button on your browser and review your post text. Otherwise please <a href="' . append_sid('viewtopic.php?p=' . $cp_row['post_id'] . '#'	. $cp_row['post_id']) .'">Click	Here</a> to review your	prior post.');
		}
		else
		{
			message_die(GENERAL_MESSAGE, 'You seem to be posting the same text in two different topics. Please do not do this, as it is called cross-posting and it	does not help our board. Cross-posting can lead	to fragmented discussions and extra work for people that want to help you. Please pick only one	forum to post your question. If	needed the topic can be	moved to a different forum later. Thanks for your help in this matter. Please <a href="' . append_sid('viewtopic.php?t=' . $cp_row['topic_id'])	.'">Click Here</a> to return to	your prior topic.');
		}
	}
}
$db->sql_freeresult($result);

The query here is simple. Go get the last three posts by this user. Other than the overhead of reading the large text field into memory this query should be extremely quick. Next, check to see if the text of any of their prior posts is essentially identical to the post they are about to enter. In the case of the browser time-out situation I described earlier the post would be 100% identical. In the case of a cross-post attempt I would expect the text to be the same, as most people type up the post and then copy/paste to enter the next. By using the soundex() function on the data I can ignore spacing and punctuation differences.

If the post text is the same, the next step is to see if the post is going into the same topic or not. A time-out warning is issued if the user is entering the same text for two posts in a row in the same topic. If they’re not posting in the same topic, then a cross-post warning is issued instead.

I am testing this now, and hope to put it into production next week.

Summary

Does this MOD impact the user experience? Yes, it does. But I am willing to do that in this case to “train” the user how to be a better board citizen. There won’t be any more post duplicates because of time-out issues, and cross-posting should be cut down as well. Both of those are desirable outcomes. If my users complain to much, I may consider reworking it. Then again, all I am doing is adding code to help them follow the rules.

Personally, I’m looking forward to no more duplicate (or triplicate) posts as well as seeing what new and creative ways people find to get around the cross-posting rejection.