First I added some javascript to the page (but only for that topic) that made the Reply and Quote buttons move away from the mouse. That made it impossible to click on the button, but you could still tab to the buttons and invoke the required code. Yesterday I switched the normal images for the buttons with the spacer.gif and sized it to zero by zero pixels, essentially making the button invisible. I also altered the tab index to -1 which according to a few sites I read makes the button disappear from the tab sequence.

Of course there are still several ways for folks to post in the topic. That’s sort of the point, to see how long it takes folks to figure out how to work around the challenges I have put in place. For example in the first version someone could disable javascript and the buttons would no longer move, giving them another way to click the button rather than using the tab key.

To continue the fun, I am looking for suggestions for other ways to challenge folks, and keep them from posting in that one topic. The key is there has to be some sort of loophole, as I’m not trying to completely lock folks out.

Any ideas?

Just a few minutes ago I was on phpbb.com and saw a post in the General Discussion with the title “Is this new home page nice?” Anyone that has been around phpbb.com for a while knows that this sort of post – even in GD – is against the rules. I figured that someone might have reported it already, but there’s no indication that such an action was taken. I decided to go ahead and report the post.

When I clicked the proper icon, here’s the message I got:

This post has already been reported.

Well. If that’s the case, why not tell me?

Reporting Posts With Feedback

I prefer my method. When a post is reported, a red “alert box” is added to that specific post, detailing when and why it was reported. It does not include who did the reporting, but that information is captured as well. Here’s what that box might look like:

This box serves a couple of purposes. First, it keeps a second (and third and fourth) person from attempting to report the post when it’s already in the queue for a moderator to review. Second, it serves as an immediate feedback to the user who posted in the wrong forum (as in the case above) and helps them learn the board rules and procedures faster.

Once the post has been acted upon, the alert box changes to show the updated status.

Suppose a person reported a post for being spam, or as in the example above for being in the wrong forum. The moderator may disagree with the assessment and decide to leave the post in the original forum. If there was no indication that this process had taken place, the post might very well be reported again. And again.

Both of the alert boxes pictured above are only shown to logged in users. Since guests can’t report posts, there’s no need for them to see this information. But any logged-in user with permissions to report a post will be told before they attempt to send the report whether it’s necessary.

Icon Explanation

The first image shown above has three icons, so I thought I would explain them briefly. The yellow flag icon allows the moderator to flag the post as “in process” or “being reviewed” for now. That means a moderator has picked up this post off of the queue and is working through the process but hasn’t decided what action to take yet. The green check icon allows the moderator to close the post and enter notes about the action(s) taken. Finally, the red X icon allows the moderator to reject the report and explain why.

If a user has accumulated a certain number of rejected reports, then their permission to report additional posts is revoked. This is to prevent someone from running around and reporting every single post they see just to be a nuisance. Over time I might also review the accept / reject ratio on post reports to determine if I want to extend an invitation to a particular user to join the moderator team.

Conclusion

It’s all about communication. The phpBB3 post report process is largely hidden. Mine is more visible. Is this a better design? I think so, but I would welcome any input, either in support of or contrary to my opinion.

With phpBB3 the moderator team can lock a post to prevent further editing. But once the original content is gone it doesn’t help. So tonight I started thinking about how and where to store post revisions in order to recover from this sort of action.

Defining the Problem

Here’s what I want to be able to do:

1. Capture the prior text of any edited post and store is somewhere
2. Track who last edited a post
3. Give moderators the ability to review the post edit history and revert back to an older version
4. Provide the ability to lock a post so further editing (except by moderators) is no longer possible

This MOD is still a work in progress so I don’t have final code to share yet. But there are a few interesting wrinkles that I thought about during the design process that I thought I would share.

Locking Posts

Locking posts is one of the easier parts. I added a status field to the phpbb_posts table. For topics there is a topic_status field that contains several different status values. I don’t need anything that complex so I called my new field post_locked and made it a tinyint(1) unsigned with a default of zero. That means that every new post that gets inserted is going to default to unlocked. Of course I added the new field to the insert statement rather than rely on the database to provide the default for me.

Once the field is present in the table I have to check it. At the beginning of viewtopic.php there are a number of authorization checks that determine which buttons / icons are displayed on the post. If the post is locked I do not display the edit button; that’s simple enough. However, what if someone creates their own URL rather than clicking the button image? In that case I have code at the top of posting.php to see if the requested function is “editpost” and then check to see if the post_locked field is set to 1 instead of 0, and if so I reject the edit attempt.

At this point I have not decided just how moderators will lock / unlock posts. One easy way would be to add a lock / unlock button on each post that moderators can see and use. However, I currently envision the locking process only being used when a board member has abused their edit permissions. That means it would be more efficient to provide my moderators a way to lock a post as they are reverting the post to a prior version.

Who Edited The Post?

With a default phpBB2 installation we don’t store who edited a post. In fact no edit history is kept at all in many cases. For example if a moderator edits a post belonging to someone else, it does not trigger the edit history. If a normal user edits a post that is the last post of a topic, that does not trigger the edit history either. The post edit history is only stored when a user edits their own post after at least one reply has been made. Since we only store the fact that a user edits their own post, there is no user_id stored. The code stores the last edit date/time as well as incrementing the overall edit count. That’s it.

To address this I added a new field called last_edit_user to the phpbb_posts table. I altered the edit history so that it runs every time a post is edited instead of only when a user edits their own post. I already have a post notes feature that records who edited the post and when. But this additional step will store the last edit user (and only the last edit user) on the post itself which means I don’t have to join out to my post notes table. I made a slight adjustment to the viewtopic.php code so that the “Last edited by…” message now includes the true user name for all edits.

What Was Changed?

This was the fun part: how do I store revisions of the post? After thinking through this and considering a number of esoteric ways to store differences in the post text I threw up my hands and decided simply to copy the entire text of the post to a new row. Why? I’m using a fraction of my disk space. I can also take advantage of the fact that in phpBB2 the post table and the post text table are separate. So here’s what I did to do that.

First I added a new field to the phpbb_posts_text table called post_version that is manditory (not null). It stores an unsigned integer value. The current version of the post text is always version zero in this table. Second, I went through the base phpBB2 code and added the following to every SQL statement that joins to the phpbb_posts_text table:

AND pt.post_version = 0

There were about 20 files that needed to have this change, but in the grand scheme of things it was a low-impact update. Once I added the column to my table and updated the code everything ran perfectly. The next step is to figure out what to store in this new field.

Tracking Versions

For most of what goes on during the post processing on a phpBB board there isn’t much to change other than the extra join column. I only ever want to search the current version. I only want to display the current version on viewtopic.php. In fact, every time I reference the phpbb_posts_text table I want only the current version… unless I am a moderator that needs to review the actual post history. Why did I make the current post version zero instead of taking the maximum number? Simple. Every post has at least one version to start with, and that version will always be zero. By ensuring that the “current” version of the post text is always stored as version 0 my join logic is extremely simple.

But what does the version number mean then? In this model, I think of the version as a number representing how many “versions ago” the text was posted. Take a post with 5 version rows in the post text table. Version 0 is the currently displayed (and search indexed) version. Version 1 is one version ago. Version 2 is two versions ago, and so on up to version 4 (the original post text) which was four versions ago compared to the current text.

How are these versions created? It’s fairly simple. In the standard phpBB2 code there is a statement that checks to see if the posting process is in edit mode or new post mode. It creates either an INSERT or UPDATE statement based on the mode. All I did was add this check:

        if ( $mode == 'editpost' )
        {
                $sql =  "UPDATE " . POSTS_TEXT_TABLE . "
                        SET     post_version = post_version + 1
                        WHERE   post_id = $post_id
                        ORDER BY post_version DESC";
                if ( !$db->sql_query($sql) )
                {
                        message_die (GENERAL_ERROR, 'Error incrementing post version', '', __LINE__, __FILE__, $sql);
                }
        }

There are a couple of interesting things here. During the edit process I increment the post version for every existing record for the post by one. This means 1 becomes 2, 2 becomes 3, and so on. This generates a SQL error because when 1 becomes 2 and 2 already exists it violates the unique primary key constraint. I added an ORDER BY clause to the update statement to ensure that I start with the end of the chain instead and it fixed that problem. By starting at the end, 3 becomes 4 before 2 becomes 3 and the constraint is never violated.

This may or may not be cross-database compatible; I don’t know which databases allow an update to have an ORDER BY clause.

After the post version numbers are incremented I always do an insert because post version zero no longer exists. That means that posts are never edited… they are always inserts. This means fewer database locks which is also a good thing.

Why Not Store Version On the Post Table?

I am going to anticipate the following question because I think it’s quite logical. In fact, I reviewed this idea myself before rejecting it in favor of what I ultimately decided on. The question?

Why not store the post version on the post table and increment it during edits, rather than using the backwards zero-based logic?

This is a good question, I think. But it’s considered bad form to ever update a database key. Once it’s set it should never change. If the post_id + post_version combination key in the post table ever got out of sync with the post text table I would have a problem. By using only the post_id as the main key and the post_version as a qualifier (and only storing the post_version in one place) I will never have that problem.

Would it have saved me some work? The only benefit is that the post table itself would store an indication of how many versions there are. But wait, I have that already. When I altered the edit process to that every edit increments the post_edit_count field I essentially got that key value. Why not use it in my join instead?

I will say it again: updating a key is a bad idea. The post edit count is informative but should not be used as a key. What happens if there is a database hiccup between the update of the posts table and the insert to the posts_text table? The edit count might be updated to 5 but I only have rows 1-4 in my post text table, and because of that the join fails. In the solution I decided to go with, I will only ever have a problem if the initial post insert process fails, otherwise I will always have a post version of zero stored in my table.

What’s Next?

At this point I have created the field needed to allow moderators to lock a post and I have written the code to check this field and prevent users from editing their post once it has been locked. I have not written the code that would allow a moderator to lock / unlock the post.

I have updated the code that tracks edits and added the last edit user to the posts table. I have also updated the code so that every edit – not just the qualifying edits as determined by the standard phpBB2 code – will trigger this code.

Finally, I have altered the posts text table so that it includes a version tag, updated all of the base phpBB2 code to reference version zero, and altered the posting process so that every edit is an insert rather than an edit to the table.

At this point I have opened a discussion with my moderator team to design the interface that they will use to interact with this new information. I need to have some way to indicate to the team that edits have been performed (already in place with the edit post notes). I need to have some way for the team to open the post history and review it. I need to allow them to decide which version of the text to revert to, and I need to provide them the option to lock the post to that text once the revert process is done.

I will be sure to post an update as the discussion moves forward. If anyone has done this before and would be willing to share your own decision ideas / process I would certainly be interested in hearing from you.

What Happened Yesterday?

One of the frequent requests that I used to see on phpbb.com was something like this:

How many visitors came to my board yesterday?

The problem I have with questions like this is that your “yesterday” is not the same as mine, unless you happen to live in the central time zone in the United States. When I wrote a MOD to do this for a client, I convinced them that rather than showing what happened “yesterday” it would be better to show what happened in the last 24 hours.

The user_lastvisit field shows the date/time that a user last logged in. This field is used to track new topics during a user session. It’s also used to drive the difference between “new” and “unread” personal messages. (A “new” message arrived since the last session. An “unread” message is one that hasn’t been read yet but arrived before the current session started.) I have altered my memberlist.php code to show when the user last visited as well.

Like most date fields in phpbb, this field is stored as int(11) rather than as a date/time field. (Other examples are the user’s registration date, the post time, new topic time. … the list goes on from there.) The content of the field is a very large integer value and is officially known as a unix timestamp.

Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.

The standard for storing date/time fields in unix timestamp is to use a signed integer rather than unsigned. This allows a developer to store negative numbers to reflect dates prior to 1970. It also has its own Y2K issue as the int(11) field will overflow in 2038. But let me get back on track for this blog post.

SQL Code for Last 24 Hours

Because of the way the user last visit time is stored, I can easily get a list of people that have visited my board in the last 24 hours with this SQL code:

select  user_id
,       username
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - 86400)
order by user_lastvisit desc

The MySQL function unix_timestamp() returns the current date and time in a unix timestamp format so I don’t have to convert anything. Since the unix timestamp is a number of seconds, and since one day has 86400 seconds, by subtracting 86400 from the current time I get the matching time from 24 hours ago. Easy stuff.

If I wanted to truly get a list of people that signed in “yesterday” then the first thing I have to do is define what yesterday means. Time zones could get involved. It could get messy. I much prefer the “last 24 hours” definition because it’s the same for everybody everywhere.

What About More Than One Day?

Sometimes I want to calculate more than one day. Instead of memorizing multiples of 86400 I simply multiply by the number of days. So if I want to count how many people have logged in for the past 7 days (as defined by 24-hour periods rather than “days”) I would do this:

select  count(user_id)
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - ( 86400 * 7 ) )

This is easy enough to do, and the code becomes “self-documenting” in a manner of speaking. I know that there are 86400 seconds in a day, and if I multiply by 7 I get a week. This is much easier to read and understand than using the number 604800.

Measuring Board Activity

About two years ago I told folks I was eagerly looking forward to the first week where my board averaged 86400 page views daily. Now that I have explained what the number is, that statement makes more sense. I was looking for the first week that I averaged a page view every second for an entire week. That happened over a year ago, and in fact my board averages over 100K page views daily at this point.

Now I am looking forward to the first week that I average 172800 page views a day. Hmm, I wonder why that is?

Related Links

• Wiki on Unix Timestamps
• Wiki on UTC
• Unix Millenium Bug

I discovered that one solution was clearly better than the other, but only if the proper index was created.

Disclaimer: I tested on phpBB2. The index that I created does not exist in a standard phpBB2, nor does it exist in a standard phpBB3 install, so I suspect this post applies to both.

Defining the Problem

Here is a partial quote of the original question in the phpBB3 MOD forum:

I want to select 5 random memebers from last 30 active users.

Simple enough, yes? First, get the last 30 members that have visited the board, then randomly select 5 of those. This could easily be done procedurally via php code, but it can also be done directly in the database with the correct SQL code.

Order By Random

The first suggestion given was this:

$start = 0;
$number = 30;
$sql = 'SELECT *
    FROM ' . USERS_TABLE . '
    ORDER BY user_lastvisit  DESC, RAND()'  ;
    $result = $db->sql_query_limit($sql, $number, $start);

With apologies to evil<3 who posted this there are a couple of things that I suggested changing. First is to avoid using the * to select every column in the table unless it’s absolutely needed. In this case, I made the assumption that the original poster was looking for a way to display a random five “recent visitors” somewhere on a page on the site. To do this doesn’t require every single bit of information about the user, just certain columns. If you select * then the entire row is returned. There are 73 columns in a standard phpBB3 users table, several of them varchar(255), and in my test installation all of the fields are mandatory. That means every single column has a value, even if it is just a space or other placeholder value. By setting up a specific list of columns to request in the query the amount of I/O is reduced. Less I/O should mean if all other things are equal the query will run faster because there are fewer bits and bytes to move around.

The other issue with the query as provided is that it’s not very efficient. It has two order by columns, one of which cannot possibly be indexed. (You can’t index something that doesn’t exist until the runtime of the query, so the rand() function result is impossible to tune.) Here is an abbreviated display for the explain plan for this query:

+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| select_type | type | possible_keys | key  | key_len | ref  | rows  | Extra                           |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| SIMPLE      | ALL  | NULL          | NULL | NULL    | NULL | 43367 | Using temporary; Using filesort |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+

This shows that no indexes will be used for this query at all, which is not good. I ran this query five times in a row on my large user table and got execution times of 0.12, 0.12, 0.12, 0.13, and 0.12 seconds.

Select Last 30 Then Random 5

As you can see from the explain data above, I have 43,367 rows in my users table right now, which is fairly large. Instead of scanning the entire table, it would be much more efficient to get the last 30 visitors in one pass and then pick five members randomly from that list. I suggested this SQL to do that:

SELECT  u.user_id
,       u.username
FROM    phpbb_users u
,       (SELECT user_id
         FROM   phpbb_users v
         ORDER BY user_lastvisit desc limit 30) v
WHERE   u.user_id = v.user_id
ORDER BY rand() limit 5;

This is an interesting technique called “inline tables” as I am creating a new table on the fly by writing SQL code inside the FROM clause. Every database I have worked with supports this technique, so it should be portable. (I do not count Microsoft Access as a real database. ) What this SQL code will do is run the inline table to return a list of 30 users, then join that virtual table to the real table by user_id (which is a unique key) and randomly select five users from the joined result set.

Is it more efficient?

I ran this query five times (as I did with the other one) and got run times of 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. It seems that it’s not really that much more effective, so is there really a clear winner?

Index Key Columns

I ran this query:

show indexes from phpbb_users

Side Note: I run my SQL directly on the database using the MySQL command line, rather than phpMyAdmin. If you use the GUI interface, then you can check for keys by looking at the appropriate screen instead of doing as I documented here.

The results of the query did not show an index on user_lastvisit which is crucial to this solution. Here is the explain plan for my query without the index:

+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key     | ref       | rows  | Extra                           |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL    | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY | v.user_id |     1 |                                 |
| DERIVED     | ALL    | NULL          | NULL    | NULL      | 43367 | Using filesort                  |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+

Notice that is also scans all 43,367 user rows. That’s okay. What isn’t okay is that it does so without the benefit of an index and it also has to do some additional work since more than one table is involved. It would seem that the first query should be more efficient since it only has one explain step and the second one has three.

However, the magic of a database indexing can fix this. The driver for this entire question is the user_lastvisit column. After creating an index on this field (which is not indexed by default in either phpBB2 or phpBB3) here is the new explain plan.

+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key            | ref       | rows  | Extra                           |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL           | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY        | v.user_id |     1 |                                 |
| DERIVED     | index  | NULL          | user_lastvisit | NULL      | 43367 |                                 |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+

Yup, still have to look at (or so the database optimizer thinks) all 43,367 users. But this time we do so with the benefit of an index. What is the impact?

The query without an index, remember, ran in 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. After creating the index I ran the same query five times and got 0.00 seconds of execution time on every trial.

Does the index help the first query? Interestingly enough, it does not. The explain plan is identical with or without the index, and the query execution times do not improve either.

However, it gets worse. It doesn’t perform as required. My query gets the last 30 users that have logged in and then randomly selects five of those. The ORDER BY clause happens after all of the select and join process is complete, so I am ordering by RAND() on at most 30 rows. The other suggestion will pick the same five users nearly every single time. Why? The secondary sort column (the “random factor”) will only come into play if two users have exactly the same last visit time (down to the second). When you have two columns in the ORDER BY clause, the first column is the primary sort and every row returned will first be sorted by that column. If there are ties in the first column then and only then will the second column be sorted.

So the first solution suggested is the worst of both worlds: not only is it slower, it is also incorrect.

Conclusion

There may be other solutions to this. I don’t mean to present this post as the ultimate answer to this question. What I hoped to accomplish with this post was to show how two solutions that look the same are not always equivalent. Subtle differences can have a huge impact on functionality.

The second lesson is that if you’re going to be asking the same question from your database over and over you should carefully consider indexing the columns used in the WHERE or ORDER BY clauses. I did some work for someone a while back (their board is one of the top ten phpBB boards on the “big boards” site). They wanted to display the top ten posters on their index. The code as written was taking over ten seconds just to run the query and then the php code / template process still had to complete. I rewrote the code and added an index on the user_posts column and the code ran in less than a hundredth of a second.

On the other hand, too many indexes can also be a problem, so don’t go out and create an index for every single column in your database. Just the ones that truly matter. In this case, it makes a substantial difference.

Implementing the Bump Warning MOD

The MOD was fairly simple to implement. It does require an extra query during the posting process. This query is used to find out if the current poster was also the last person to post in the selected topic. If so, it checks the post time to see if 24 hours have expired. If not, then it sets a value for a template switch which turns on the red warning label on the posting screen. Here’s the code from posting.php.

// BEGIN Bump Warning 1.0.0 (www.phpBBDoctor.com)
if ($mode == 'reply')
{
        $sql = 'SELECT  p.post_id
                ,       p.poster_id
                ,       p.post_time
                FROM    ' . POSTS_TABLE . ' p
                ,       ' . TOPICS_TABLE . ' t
                WHERE   t.topic_id = ' . $topic_id . '
                AND     p.post_id = t.topic_last_post_id+0';

        if (!($result = $db->sql_query($sql)))
        {
                message_die(GENERAL_ERROR, 'Invalid cursor getting last poster', '', '', '', $sql);
        }
        $check = $db->sql_fetchrow($result);
        $last_post_id = $check['post_id'];
        $db->sql_freeresult($result);

        if ( ($check['poster_id'] == $userdata['user_id']) && ($check['post_time'] > time() - (86400)) )
        {
                $template->assign_block_vars('switch_bump_warning', array(
                        'U_EDIT_PRIOR_POST' => append_sid("posting.$phpEx?mode=editpost&p=$last_post_id")
                        ));
        }
}
// END Bump Warning 1.0.0 (www.phpBBDoctor.com)

Then of course in the template file I have the switch defined.

<!-- BEGIN switch_bump_warning -->
<tr>
<td class="favrow_red" width="22%" height="35"><span class="gen"><strong>Bump Warning</strong></span></td>
<td class="favrow_red" width="78%"><span class="gen">It appears that you were the last person to post in this topic and 24 hours has not yet elapsed. Please do not reply to your post unless you have new information to add. You may also <a href="{switch_bump_warning.U_EDIT_PRIOR_POST}" class="gen">edit your prior post</a> since nobody has replied yet. Thanks.</td>
</tr>
<!-- END switch_bump_warning -->

This is all well and good, but it seems that some people can be blind. Somehow the red banner across the top of the screen isn’t enough to let them know they’re doing something that I don’t want them to do. I even give them a link to edit the prior post instead of adding a new one…

Since I have asked nicely and they’re still bumping posts, what do I do next?

Duplicate Post Prevention

There are two kinds of post bumps that I typically see. The first kind is what I have described above where the person is simply ignoring my request to not do what he or she is about to do. The second kind is also common and is not really something I can blame a user for. At times the Internet gets slow, and the posting process times out. It may also be that there are several hundred people watching a topic and it takes time to send those emails. (Something else I want to modify; I want to set up a job-queue to handle this process so the poster doesn’t pay a penalty for that during their posting process.) The bottom line is that sometimes a post times out, so the user resubmits. And resubmits. And when they’re done, there are two (or three or more) posts in a row that contain the same text.

So tonight I am starting to test a new feature. I am supplementing my bump “warning” with a bump “rejection” as well. The bump rejection does not prevent a legitimate post bump, but it will catch users that enter the same post twice due to the time-out issue I described above. As an added bonus, this new code will also detect a cross-posted topic and prevent that as well.

To make this change I opened includes/functions_post.php and added code to the Flood Control section. As this is very preliminary I am not using any language strings. That being said, here’s the code.

// Cross-post capture
$sql = 'select	post_text
	,	p.post_id
	,	p.topic_id
	from	' . POSTS_TEXT_TABLE . ' pt
	,	' . POSTS_TABLE	. ' p
	where	p.poster_id = '	. $userdata['user_id'] . '
	AND	p.post_id = pt.post_id
	ORDER BY p.post_id desc	limit 3';

if (!($result =	$db->sql_query($sql) ) )
{
	message_die(GENERAL_ERROR, 'Invalid cursor while checking for cross posts');
}

while ($cp_row = $db->sql_fetchrow($result))
{
	if (soundex($post_message) == soundex($cp_row['post_text']))
	{
		if ($topic_id == $cp_row['topic_id'])
		{
			message_die(GENERAL_MESSAGE, 'You are attempting to post the same item twice. It is possible that your browser timed-out on your prior post submission.	If you think this is an	error, please click the	"back" button on your browser and review your post text. Otherwise please <a href="' . append_sid('viewtopic.php?p=' . $cp_row['post_id'] . '#'	. $cp_row['post_id']) .'">Click	Here</a> to review your	prior post.');
		}
		else
		{
			message_die(GENERAL_MESSAGE, 'You seem to be posting the same text in two different topics. Please do not do this, as it is called cross-posting and it	does not help our board. Cross-posting can lead	to fragmented discussions and extra work for people that want to help you. Please pick only one	forum to post your question. If	needed the topic can be	moved to a different forum later. Thanks for your help in this matter. Please <a href="' . append_sid('viewtopic.php?t=' . $cp_row['topic_id'])	.'">Click Here</a> to return to	your prior topic.');
		}
	}
}
$db->sql_freeresult($result);

The query here is simple. Go get the last three posts by this user. Other than the overhead of reading the large text field into memory this query should be extremely quick. Next, check to see if the text of any of their prior posts is essentially identical to the post they are about to enter. In the case of the browser time-out situation I described earlier the post would be 100% identical. In the case of a cross-post attempt I would expect the text to be the same, as most people type up the post and then copy/paste to enter the next. By using the soundex() function on the data I can ignore spacing and punctuation differences.

If the post text is the same, the next step is to see if the post is going into the same topic or not. A time-out warning is issued if the user is entering the same text for two posts in a row in the same topic. If they’re not posting in the same topic, then a cross-post warning is issued instead.

I am testing this now, and hope to put it into production next week.

Summary

Does this MOD impact the user experience? Yes, it does. But I am willing to do that in this case to “train” the user how to be a better board citizen. There won’t be any more post duplicates because of time-out issues, and cross-posting should be cut down as well. Both of those are desirable outcomes. If my users complain to much, I may consider reworking it. Then again, all I am doing is adding code to help them follow the rules.

Personally, I’m looking forward to no more duplicate (or triplicate) posts as well as seeing what new and creative ways people find to get around the cross-posting rejection.

The banner system is fairly complex, but at the most basic level there is a cron (scheduled) job that periodically decrements the sponsor view balance. Once the view balance hits zero, the sponsor’s banners are deactivated until they pay for another round. The code is quite simple:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  view_balance - ' . $decrement_views . '
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

The value for $decrement_views is assigned earlier in the loop. The definition for the view_balance column is an unsigned integer (mediumint specifically) so it will not allow negative values. On my old server this worked perfectly. If $decrement_views was greater than view_balance the sponsor view balance was set to zero. My old server was running MySQL 4.1.

My new server is running 5.0, and this same code did not work. Unfortunately it did not generate a syntax or other runtime error. Instead it did the math wrong.

Integer Storage in MySQL

Before I talk more about the bug I think I should talk about how computers store numbers. This is not specific to MySQL, it can affect any system that stores numeric values. When I store a number I have a choice of adding the unsigned attribute. In MySQL it takes the following format. The first will store a tiny integer without a sign, and the second will store a tiny integer with a sign.

create table dave (new_column tinyint unsigned);

create table dave (new_column tinyint);

What is the difference? When numbers are stored they take space. A tinyint column in MySQL can store values from -128 to 127. An unsigned tinyint can store values from 0 to 255. How does this work, and why are the numbers different in each case?

A tinyint is stored in one byte or eight bits of information. With eight bits I have a range of 0000 0000 to 1111 1111. With an unsigned value I can use all eight bits for my number, so 0000 0000 = 0 and 1111 1111 is 1 + 2 + 4 + 8 + 16 + 32 + 64 + 128, or – if you do the math – 255. That’s how an unsigned tiny integer column can store a value ranging from 0 to 255. If, however, I want to use a signed value, the first bit becomes an indication of whether the value is negative or not. That means I only have seven bits left to determine the value. 0111 1111 becomes 1 + 2 + 4 + 8 + 32 + 64 which is 127, or the maximum signed value that can be stored in a signed tiny integer field. What happens when the eighth bit gets flipped to a 1? That’s an indication that the number is negative instead of positive. So while both signed and unsigned values take the same amount of space, a signed value is one order of magnitude smaller because the most significant bit (the leading bit) is used to indicate the sign of the value that is stored.

Put another way: a signed tinyint has seven available bits and therefore can store 2⁷-1 or 127 as the maximum value. An unsigned tinyint has eight available bits and therefore can store 2⁸-1 or 255 as the maximum value. Suppose I am looking at a number in memory and the bit values are 1000 0001. What is the value represented by these bits?

The fact is I can’t make that determination until I know if the value is signed or not. If the value is unsigned, the number represented by 1000 0001 is 129. If it’s signed, it gets complicated but the value returned would be -127. Keep in mind that I am using tinyint example values here.

How Unsigned Math Broke My Sponsor System

In my banner system I don’t try to track the page views down to exactly zero. A sponsor will pay for two million page views at a time. If they actually use two million and twelve I am not going to complain about the few extra views. So my system is designed to allow each sponsor more views in the interest of good will but mostly for system performance. In my system, each sponsor’s view balance is only updated once an hour. Let me repeat the code that I showed above:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  view_balance - ' . $decrement_views . '
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

First (not shown) I get a total of the banner views that have accumulated over the past hour and store them into the $decrement_views variable in my php script. Next I execute the SQL script shown above for each sponsor with an active banner. Suppose sponsor number 12 has 1000 views left and they used 300 in the last hour. The SQL code resolves to this:

UPDATE phpbb_sponsors
SET view_balance = 1000 - 300
WHERE sponsor_id = 12;

After this statement is executed the sponsor has a balance of 700 views left. Suppose the same sponsor has 100 views in their balance and they used 300 more during the last hour. The SQL ends up looking like:

UPDATE phpbb_sponsors
SET view_balance = 100 - 300
WHERE sponsor_id = 12;

When 300 is subtracted from 100 it results in a negative number. Under my old version of MySQL that number was set to zero since the column is defined as unsigned and is not capable of storing a negative value. This is what broke during the upgrade to MySQL 5.

MySQL Bug Explained

The newer version of MySQL did the math as signed, which allowed the value to go negative, and then stored the results in the unsigned field. You might start to see the problem now. Instead of setting the sponsor view balance to zero for any negative result, the sponsor view balance was set to the maximum value that it could possibly hold because of the overflow. In this case, the view balance got set to 16,777,215 instead of zero. What is significant about that number? Here is a quote from the MySQL web page where it details the values that can be stored for any particular numeric column type…

MEDIUMINT[(M)] [UNSIGNED] [ZEROFILL]
A medium-sized integer. The signed range is -8388608 to 8388607. The unsigned range is 0 to 16777215.

In a nutshell, the old version of MySQL caught the overflow exception and set the value to zero. The newer version of MySQL did not handle the overflow and instead let the signed value stored the unsigned negative value. I’m sure that would have made my sponsors happy, but it certainly wasn’t how things were intended to work.

Fixing The Problem

There is a good lesson to be learned here. I was lazy in the way I wrote my earlier code. Rather than check to see if the number of accumulated views was higher than the balance remaining and handling that exception with code, I relied on the fact that MySQL would not (should not) store a negative result in an unsigned field. When the database behavior changed (it has been recognized as a bug by MySQL according to my research) my system broke.

I have fixed the SQL by using a case statement so that this error will never occur for me again. Here is the revised SQL:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  case
			when ' . $decrement_views . ' > view_balance then 0
			else view_balance - ' . $decrement_views . '
			end
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

This updated code uses a case statement structure to check to make sure that the remaining balance is larger than the value to be decremented. If it is not, the value is simply set to zero.

Finally, now that I’ve explained signed versus unsigned it makes the following cartoon from xkcd.com more meaningful, doesn’t it?

Related Links

• MySQL Numeric Types Explained
• Unsigned and Signed Integers, an article I found on the Internet with more details on signed versus unsigned storage, it’s short and a very easy read

On localhost works but not works in linux server.

Localhost: all pages works
Sever linux: none pages works

So basically they’re telling me that the MOD works on their localhost installation, but once they upload it to their linux server “none pages works” which, I guess, is a bit of a problem.

But without my ESP module addon for the PM system, I am unable to determine exactly what the problem is, so I was forced to ask them for a specific issue. So folks, if you have a problem with my code, tell me what it is. If it’s supposed to do X and it does Y instead, tell me that. If it’s supposed to do X and it’s not doing anything, tell me that.

But don’t tell me “it doesn’t work” and expect much help.

Where is this going? Based on the remarks made earlier tonight I worked out a really quick and easy MOD for phpBB2 (and the idea would work just as well for phpBB3). It’s a few bits of code that allow users to set a fixed width for the board (by pixel count) or opt for a full screen display. It took about an hour from initial concept to execution.

Building the User Selectable Board Width MOD

Step one is creating a place to store this information.

alter table phpbb_users
add user_board_width smallint(5) unsigned not null default 0;

I opted to use smallint(5) because the highest value I anticipate is 1600. The MySQL smallint data type goes up to 65535 for an unsigned value, so that should be plenty. The default value is defined as 0, which my code will interpret as full screen. Any other value will be used as the screen width.

Step 2 is to let the user enter something into that new value. Here’s what that looks like:

The code to build the selector and handle the user input value is all in includes/usercp_register.php which is the dual-purpose register and edit profile program.

What I did next I will call step 2.5 since it would not really be necessary. I added admin control that lets the board owner set the default width in case the user doesn’t pick anything. The value for that is stored in the phpbb_config table and therefore added to the $board_config array.

Step 4 is to collect the user preference and pass it to the template. That process takes place in includes/page_header.php like this:

'TABLE_WIDTH' => (intval($userdata['user_board_width']) == 0 ? $board_config['default_board_width'] : intval($userdata['user_board_width'])),

As you can see, if the $userdata array contains a zero value for the board width, the default value entered by the board admin will be used. Otherwise the user profile option is converted to an integer and sent to the template. That’s all the coding that was required. The template changes are even easier.

subSilver Template Changes

For subSilver the final step is very simple. Open the overall_header and find the top table that contains the rest of the page structure. It starts out looking like this:

<table width="100%" cellspacing="0" cellpadding="0" border="0" align="center">

Change it to this, and you’re done:

<table width="{TABLE_WIDTH}" cellspacing="0" cellpadding="0" border="0" align="center">

Fluid versus Fixed Design Debate

The debate over fixed versus fluid design can be as contentious as the debate comparing Mac to PC or IE to FF. Well, perhaps not that bad, but I’ve seen it get close. As a user, I perfer fluid sites so I can resize the window to fit my preferences. As a designer I think fixed width designs are easier to create. Making something scale well from 800 pixels all the way up to 1600 pixels (or beyond) is quite a challenge. When I was looking for templates for one of my blogs, I found that the ratio of fixed to fluid templates was probably about 5:1 (five fixed for every one fluid). And many of the fluid templates were … less than useful.

The subSilver template is a great example of a style that works both ways. For any style that fits the same pattern, having this simple MOD installed puts the choice back into the hands (and mouse) of your user. I think that’s a good thing.

I will be writing up the full MOD install and posting it in my MOD Catalog so that it’s available on this site.

The problem with this and other anti-spam techniques is that it’s based on words rather than content. This may seem like splitting hairs… after all, isn’t my content made up of words? Yes, yes it is. And that’s the problem. Confused yet? I hope so, because it gets worse from here.

Words are Words, Content is Combinations of Words

Simply put, to catch and prevent spam posts in any sort of programatic fashion the code has to be smart enough to understand content and not just words. Examine for a moment the following two phrases:

Fruit flies like a banana.
Time flies like an arrow.

I didn’t make these up; the combination of these two phrases appears quite often in discussions about language or pattern recognition. The two phrases are nearly identical. Each has five words. In fact, the middle three words in each phrase are essentially identical. Yet the phrases mean something completely different. In one phrase the word “flies” is a noun (an object) and in the other it’s a verb. The word “like” is used in two different ways. If I were to examine these two sentences word by word I would probably conclude that there is a high degree of correlation between the two. In fact, there is very little.

Here’s another example that I saw recently. I could not remember it exactly, but it was something like this:

Bank of New Zealand floods customer inboxes.
New Zealand river floods, overflows bank.

Again, if I were to look at an individual word comparision these two sentences look very similar. They each contain the words (or forms of the words) “new”, “zealand”, “river”, “flood”, and “bank” in the sentence. When I first saw the example (which I cannot find at the moment) there were some other similar words as well. In order to properly differentiate these two sentences I have to go beyond word analysis and do a context or content analysis.

And what about this headline:

Hacker penetrates Paris Hilton

Is that an article about a security flaw in a hotel network? Or a pornography video?

Unstructured Data

Unstructured data analysis is becoming more and more interesting to corporations for a wide variety of reasons. None of them are related to fighting spam. Other than hiring an army of readers, how is a company to know what is being said about it on the web? There are sites like epinions.com and resellerratings.com that allow people to log on and post reviews about various products. There are newsgroups hosted by Yahoo! and Google where people can log on and post complements or complaints. There are blogs, discussion boards, and “sucks” sites. There are legitimate news articles or press releases. There are social networking sites. In short, there is a flood (heh) of information on the web, and very little of it is structured. If programmers at billion dollar companies are struggling with how to manage that information, what are we as phpBB MOD authors supposed to do?

I have often talked about my “big board” on this site. The board is an independent discussion board related to the products from a company named Business Objects (which recently was acquired by SAP). One of the products that Business Objects bought in 2007 was a company called Inxight which was a result of yet another Xeroc PARC research product. This product is designed to process unstructured data and perform content recognition. They have a fairly high-level demo online; I have included a link at the end of this post. The demo is light on specifics but it does show how the product can scan unstructured data like a press release and extract the important concepts and data points.

Anti-spam Application

And now I am finally getting back to the idea presented in the first paragraph: can we use word analysis to combat spammers on our boards? I think that the answer is “not yet” because we don’t have algorithms that are sophisticated enough to manage context. There are a number of anti-spam MODs in various stages that look at words, but to my knowledge there aren’t any that analyze the context of the words. A collection of words taken separately might indicate spam, but when reviewed in context they might be a perfectly valid post.

In other words, it’s not enough to identify words, I also need to identify how those words are used.

Related Posts

Another application for content analysis is a “related posts” MOD. There are a number of these for phpBB2. One I read used the phpBB2 search tables to identify common words by frequency across topics. Another used a special database index on the topic title only. To be honest, if posts are related because of common word usage, in my opinion they are flawed. If the posts are related because of relevance… that’s something I would be interested in. I did some experiments with a related posts MOD of my own and ultimately never completed the project due to my lack of satisfaction with the algorithms I could find or come up with on my own. I would like to revisit this idea again in the future.

Conclusion

The bottom line is that blogs and boards have one very important thing in common: the data is nearly completely unstructured. I say “nearly” because with blogs we have categories, and a board has a specific category -> forum -> topic hierarchy in place. But outside of that, the content provided by a board post may have nothing to do with anything else on the board. Does that make it spam? It’s hard to say. That’s why we still need good moderators for our boards.

Time flies like an arrow. Fruit flies like a banana. My board can’t tell the difference, can yours?

Related Links

• Wiki on Bayesian Filters
• Turning Unstructured Text into Insight