What Happened Yesterday?

One of the frequent requests that I used to see on phpbb.com was something like this:

How many visitors came to my board yesterday?

The problem I have with questions like this is that your “yesterday” is not the same as mine, unless you happen to live in the central time zone in the United States. When I wrote a MOD to do this for a client, I convinced them that rather than showing what happened “yesterday” it would be better to show what happened in the last 24 hours.

The user_lastvisit field shows the date/time that a user last logged in. This field is used to track new topics during a user session. It’s also used to drive the difference between “new” and “unread” personal messages. (A “new” message arrived since the last session. An “unread” message is one that hasn’t been read yet but arrived before the current session started.) I have altered my memberlist.php code to show when the user last visited as well.

Like most date fields in phpbb, this field is stored as int(11) rather than as a date/time field. (Other examples are the user’s registration date, the post time, new topic time. … the list goes on from there.) The content of the field is a very large integer value and is officially known as a unix timestamp.

Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.

The standard for storing date/time fields in unix timestamp is to use a signed integer rather than unsigned. This allows a developer to store negative numbers to reflect dates prior to 1970. It also has its own Y2K issue as the int(11) field will overflow in 2038. But let me get back on track for this blog post.

SQL Code for Last 24 Hours

Because of the way the user last visit time is stored, I can easily get a list of people that have visited my board in the last 24 hours with this SQL code:

select  user_id
,       username
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - 86400)
order by user_lastvisit desc

The MySQL function unix_timestamp() returns the current date and time in a unix timestamp format so I don’t have to convert anything. Since the unix timestamp is a number of seconds, and since one day has 86400 seconds, by subtracting 86400 from the current time I get the matching time from 24 hours ago. Easy stuff.

If I wanted to truly get a list of people that signed in “yesterday” then the first thing I have to do is define what yesterday means. Time zones could get involved. It could get messy. I much prefer the “last 24 hours” definition because it’s the same for everybody everywhere.

What About More Than One Day?

Sometimes I want to calculate more than one day. Instead of memorizing multiples of 86400 I simply multiply by the number of days. So if I want to count how many people have logged in for the past 7 days (as defined by 24-hour periods rather than “days”) I would do this:

select  count(user_id)
from    phpbb_users
where   user_lastvisit >= (unix_timestamp() - ( 86400 * 7 ) )

This is easy enough to do, and the code becomes “self-documenting” in a manner of speaking. I know that there are 86400 seconds in a day, and if I multiply by 7 I get a week. This is much easier to read and understand than using the number 604800.

Measuring Board Activity

About two years ago I told folks I was eagerly looking forward to the first week where my board averaged 86400 page views daily. Now that I have explained what the number is, that statement makes more sense. I was looking for the first week that I averaged a page view every second for an entire week. That happened over a year ago, and in fact my board averages over 100K page views daily at this point.

Now I am looking forward to the first week that I average 172800 page views a day. Hmm, I wonder why that is?

Related Links

• Wiki on Unix Timestamps
• Wiki on UTC
• Unix Millenium Bug

The banner system is fairly complex, but at the most basic level there is a cron (scheduled) job that periodically decrements the sponsor view balance. Once the view balance hits zero, the sponsor’s banners are deactivated until they pay for another round. The code is quite simple:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  view_balance - ' . $decrement_views . '
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

The value for $decrement_views is assigned earlier in the loop. The definition for the view_balance column is an unsigned integer (mediumint specifically) so it will not allow negative values. On my old server this worked perfectly. If $decrement_views was greater than view_balance the sponsor view balance was set to zero. My old server was running MySQL 4.1.

My new server is running 5.0, and this same code did not work. Unfortunately it did not generate a syntax or other runtime error. Instead it did the math wrong.

Integer Storage in MySQL

Before I talk more about the bug I think I should talk about how computers store numbers. This is not specific to MySQL, it can affect any system that stores numeric values. When I store a number I have a choice of adding the unsigned attribute. In MySQL it takes the following format. The first will store a tiny integer without a sign, and the second will store a tiny integer with a sign.

create table dave (new_column tinyint unsigned);

create table dave (new_column tinyint);

What is the difference? When numbers are stored they take space. A tinyint column in MySQL can store values from -128 to 127. An unsigned tinyint can store values from 0 to 255. How does this work, and why are the numbers different in each case?

A tinyint is stored in one byte or eight bits of information. With eight bits I have a range of 0000 0000 to 1111 1111. With an unsigned value I can use all eight bits for my number, so 0000 0000 = 0 and 1111 1111 is 1 + 2 + 4 + 8 + 16 + 32 + 64 + 128, or – if you do the math – 255. That’s how an unsigned tiny integer column can store a value ranging from 0 to 255. If, however, I want to use a signed value, the first bit becomes an indication of whether the value is negative or not. That means I only have seven bits left to determine the value. 0111 1111 becomes 1 + 2 + 4 + 8 + 32 + 64 which is 127, or the maximum signed value that can be stored in a signed tiny integer field. What happens when the eighth bit gets flipped to a 1? That’s an indication that the number is negative instead of positive. So while both signed and unsigned values take the same amount of space, a signed value is one order of magnitude smaller because the most significant bit (the leading bit) is used to indicate the sign of the value that is stored.

Put another way: a signed tinyint has seven available bits and therefore can store 2⁷-1 or 127 as the maximum value. An unsigned tinyint has eight available bits and therefore can store 2⁸-1 or 255 as the maximum value. Suppose I am looking at a number in memory and the bit values are 1000 0001. What is the value represented by these bits?

The fact is I can’t make that determination until I know if the value is signed or not. If the value is unsigned, the number represented by 1000 0001 is 129. If it’s signed, it gets complicated but the value returned would be -127. Keep in mind that I am using tinyint example values here.

How Unsigned Math Broke My Sponsor System

In my banner system I don’t try to track the page views down to exactly zero. A sponsor will pay for two million page views at a time. If they actually use two million and twelve I am not going to complain about the few extra views. So my system is designed to allow each sponsor more views in the interest of good will but mostly for system performance. In my system, each sponsor’s view balance is only updated once an hour. Let me repeat the code that I showed above:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  view_balance - ' . $decrement_views . '
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

First (not shown) I get a total of the banner views that have accumulated over the past hour and store them into the $decrement_views variable in my php script. Next I execute the SQL script shown above for each sponsor with an active banner. Suppose sponsor number 12 has 1000 views left and they used 300 in the last hour. The SQL code resolves to this:

UPDATE phpbb_sponsors
SET view_balance = 1000 - 300
WHERE sponsor_id = 12;

After this statement is executed the sponsor has a balance of 700 views left. Suppose the same sponsor has 100 views in their balance and they used 300 more during the last hour. The SQL ends up looking like:

UPDATE phpbb_sponsors
SET view_balance = 100 - 300
WHERE sponsor_id = 12;

When 300 is subtracted from 100 it results in a negative number. Under my old version of MySQL that number was set to zero since the column is defined as unsigned and is not capable of storing a negative value. This is what broke during the upgrade to MySQL 5.

MySQL Bug Explained

The newer version of MySQL did the math as signed, which allowed the value to go negative, and then stored the results in the unsigned field. You might start to see the problem now. Instead of setting the sponsor view balance to zero for any negative result, the sponsor view balance was set to the maximum value that it could possibly hold because of the overflow. In this case, the view balance got set to 16,777,215 instead of zero. What is significant about that number? Here is a quote from the MySQL web page where it details the values that can be stored for any particular numeric column type…

MEDIUMINT[(M)] [UNSIGNED] [ZEROFILL]
A medium-sized integer. The signed range is -8388608 to 8388607. The unsigned range is 0 to 16777215.

In a nutshell, the old version of MySQL caught the overflow exception and set the value to zero. The newer version of MySQL did not handle the overflow and instead let the signed value stored the unsigned negative value. I’m sure that would have made my sponsors happy, but it certainly wasn’t how things were intended to work.

Fixing The Problem

There is a good lesson to be learned here. I was lazy in the way I wrote my earlier code. Rather than check to see if the number of accumulated views was higher than the balance remaining and handling that exception with code, I relied on the fact that MySQL would not (should not) store a negative result in an unsigned field. When the database behavior changed (it has been recognized as a bug by MySQL according to my research) my system broke.

I have fixed the SQL by using a case statement so that this error will never occur for me again. Here is the revised SQL:

$sql = 'UPDATE  ' . SPONSORS_TABLE . '
	SET     view_balance =  case
			when ' . $decrement_views . ' > view_balance then 0
			else view_balance - ' . $decrement_views . '
			end
	WHERE   sponsor_id = ' . $sponsor_data[$i]['sponsor_id'];

This updated code uses a case statement structure to check to make sure that the remaining balance is larger than the value to be decremented. If it is not, the value is simply set to zero.

Finally, now that I’ve explained signed versus unsigned it makes the following cartoon from xkcd.com more meaningful, doesn’t it?

Related Links

• MySQL Numeric Types Explained
• Unsigned and Signed Integers, an article I found on the Internet with more details on signed versus unsigned storage, it’s short and a very easy read

MySQL offers an interesting function called LAST_INSERT_ID() that – once I figured out the proper syntax – provided exactly what I needed for my accurate and efficient banner system. What does it do? Simply put, it provides a shortcut to return the result of a previous SQL statement without having to worry about intervening updates. Before I explain the function, I want to share a bit of the database design and how the process will eventually work.

Page Count Math

I have a table called phpbb_views. The structure of this table is very simple: it only contains a single column named view_ctr. The field is currently defined as int(11) unsigned which means it can go up to a really large number. What can I say… I’m optimistic. I have had code in place for years that increments this value on every page view. Previously I had the update statement running inside the includes/page_tail.php code. For the banner system I moved that logic into the includes/page_header.php code instead.

Let me assume that I have 10 banners that are all equally weighted. I want to ensure that these banners are displayed in sequential order, from one to ten and then starting over again. There is a perfect function for this, and you might have seen it if you’ve looked at how phpBB2 generates alternating row colors for tables. The syntax looks like this:

($var1 % $var2)

The result is the remainder of $var1 divided by $var2. Note that both $var1 and $var2 are converted to integer values (if needed) prior to the operation. How can this help me?

A sequential number (like page view counter values) will increment by one, starting at some number, and extending to infinity. What I need is a rotating sequence of numbers that starts at 1 and goes to 10 and then starts over again. That’s exactly what the % operator will do for me. All I have to do is use my page view counter as $var1 and 10 (the number of banners) as $var2 and here’s what the numbers look like:

+-----+--------------+
| 100 |            0 |
| 101 |            1 |
| 102 |            2 |
| 103 |            3 |
| 104 |            4 |
| 105 |            5 |
| 106 |            6 |
| 107 |            7 |
| 108 |            8 |
| 109 |            9 |
| 110 |            0 |
+-----+--------------+

If you notice the column on the left has an increasing sequence, while the column on the right starts with zero and increments up to nine and then starts over again. This is because 100 / 10 has no remainder, while 101 / 10 has a remainder of one, and so on. This is perfect! Now all I have to do is ensure that I will never skip a number and I have a guaranteed way to pick the next (and correct) banner to display. If I have a zero-based array index these numbers are already perfect. If my array index starts at one then all I have to do is change the formula to ($var1 % $var2) + 1 and I am all set.

How the MySQL LAST_INSERT_ID() Function Works

I will jump right to the syntax of my query and then explain it next:

UPDATE phpbb_views
SET view_ctr = LAST_INSERT_ID(view_ctr+1) ;

This statement accomplishes two things at the same time. First, it updates the view_ctr column using the provided expression (view_ctr + 1). Second, the LAST_INSERT_ID() function makes a note of the resulting value and stores it as a part of my session. In a sense, it is doing the update and the select all at the same time! This eliminates any potential issue with statements appearing out of order. It also eliminates any need to perform any locking on the rows or the table.

To retrieve the session value I issue this command:

SELECT LAST_INSERT_ID() ;

That is simple, isn’t it? As per the MySQL documentation this query does not even reference a table. It pulls the last affected value from the session variable and provides it to me. If the view_ctr value was 110 and I run the first command the new value for view_ctr is 111 and that value is also cached for my session. When I retrieve the value using the second query I get 111.

All of my problems are solved. I can use this technique without worrying about a deadlock and without worrying about missing values. It didn’t take long to set up my code with this function. But is it fast?

On average that query takes about 0.00015 seconds. That’s not too bad. What’s even better is that the query to set the new value as well as the one required to update the banner counters both are equally as efficient.

My final banner system provides me with the following features:

• Banners can have start dates or expiration dates
• Advertisers can have multiple banners on file
• Advertisers can “weight” banners so they appear more (or less) frequently
• Page views are evenly distributed across all active banners according to weight

There are many more features in place, but the last bullet point is the result of the work done to get this far. I will talk more about the table design and how I completed my banner system in my next post.

Problem Defined

I needed a way to set up a sequence of banners and then ensure that nobody ever got skipped. If I said that each banner was going to be displayed an equal number of times per day, then I had better be sure that I delivered that. Random number generators are historically not very random and therefore I could not use that approach. Having a “next banner to be displayed” record in a database would work, but the lock for update / update / select / release lock process might result in a deadlock situation, and certainly would not perform very well. I decided I needed to do some research and figure out the best way to manage my banner rotation.

What I found was really interesting, and has been working amazingly well for several months now. Unfortunately the technique is unique to MySQL so I can’t plan on releasing this as a MOD anytime soon.

Random Numbers versus Incrementing Page View Counter

I had decided that I did not want to use any sort of random number generator for showing banners. My preliminary testing showed that it was quite frankly a horrible solution. There was always one banner that was displayed a lot more than any of the others. Perhaps over time it would have become more even, but after 100 trials and 10 banners it seemed that there was always one banner in the 20+ range when they should have all been (if equally weighted) around 10. That was unacceptable and I quickly eliminated this from consideration.

Next I decided to leverage a sequential number that I already had. In page_tail.php I had a very simple query that was responsible for updating a page view counter. By using the mod() function on the newly assigned page counter number I could rely on a continuing series of numbers that would never skip values.

Or could I?

On a very busy board there might be multiple updates occuring nearly at the same time. If I were to use this technique, as shown by these two simple SQL statements:

UPDATE phpbb_page_table SET page_counter = page_counter + 1;
SELECT page_counter FROM phpbb_page_table;

It is entirely possible (and fact even quite likely) that someone else could have updated the page counter value before I selected my result row. That meant that even if the code looked like what I typed above the reality is that it could get executed like this:

UPDATE phpbb_page_table SET page_counter = page_counter + 1;
UPDATE phpbb_page_table SET page_counter = page_counter + 1;
UPDATE phpbb_page_table SET page_counter = page_counter + 1;
SELECT page_counter FROM phpbb_page_table;
UPDATE phpbb_page_table SET page_counter = page_counter + 1;
SELECT page_counter FROM phpbb_page_table;
SELECT page_counter FROM phpbb_page_table;
SELECT page_counter FROM phpbb_page_table;

Note that there are still four “update” statements and four “select” statements, but they’re not guaranteed to execute in the proper order since each request is coming from a different page. Of course each page only sees one SELECT and one UPDATE statement but there is no way to ensure that someone else’s page isn’t running the same queries at essentially the exact same time that I am. The queries are sent to the database from all pages at the same time. Any page logic that requires an absolutely isolated query execution sequence is going to potentially fail in a shared user environment. It might not fail every time, but it could potentially fail. That’s a problem.

What is the result of having the queries out of order in the above example? Banner number 1 and 2 get skipped because the update runs with no select. Banner number 3 is displayed once, and banner number 4 is displayed three times because there are three selects following a single update. The net result is that I am going to potentially see the same issue here that I saw with my random number system… banners were not going to be guaranteed to come out with an even distribution of page views.

Select … For Update

To avoid having multiple updates run before I can get my selected value returned I could consider using a lock statement of some kind. That way when the select statement is issued it is guaranteed that nobody else has updated that row. After the select is performed then the lock would be released. Here’s how this would have worked out if I had wanted to try this solution.

MySQL provides the “Select FOR UPDATE” syntax. This will use page or row locking (whatever is available based on the engine in use) instead of locking the entire table. In the scenario shown in the prior section I ran the update and then selected the resulting value. This time the logic will be reversed, as shown here:

SELECT page_counter FROM phpbb_page_table FOR UPDATE;
UPDATE phpbb_page_table SET page_counter = page_counter + 1;

The “FOR UPDATE” clause is used to prevent anyone else from updating the row until I get my value. Once I get the value, I increment the counter for the next person. The lock has to be an exclusive lock otherwise two pages would end up competing for the update.

I am really hesitant to issue application locks for these and other reasons. Obviously it’s fine if the database issues locks as a result of commands that I execute, but having specific code in my php files to issues locks? Not a good idea. To be honest, I didn’t even try to prototype or test this solution because it would have been horrible. What if a page dies in the middle of the lock and never releases it?

To summarize so far:

• Random numbers are efficient but not guaranteed accurate
• Sequential counters are efficient but not guaranteed accurate
• Sequential counters with locks are accurate but not efficient and could lead to a deadlock situation

It turns out that while I was researching lock statements on mysql.com I found this buried at the very bottom of the page:

The preceding description is merely an example of how SELECT … FOR UPDATE works. In MySQL, the specific task of generating a unique identifier actually can be accomplished using only a single access to the table:

UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();

The SELECT statement merely retrieves the identifier information (specific to the current connection). It does not access any table.

Hm. That sounds very interesting. In fact, it’s exactly the solution that I needed for my banner system to be both 100% accurate and efficient at the same time. I’ll talk more about how the LAST_INSERT_ID() function works in my next post.

To encode an IP use this expression:

mysql> select conv(inet_aton('127.0.0.1'),10,16);
+------------------------------------+
| conv(inet_aton('127.0.0.1'),10,16) |
+------------------------------------+
| 7F000001                           |
+------------------------------------+
1 row in set (0.00 sec)

To decode an IP address use this expression:

mysql> select inet_ntoa(conv('7F000001',16,10));
+-----------------------------------+
| inet_ntoa(conv('7F000001',16,10)) |
+-----------------------------------+
| 127.0.0.1                         |
+-----------------------------------+
1 row in set (0.01 sec)

The functions used here are conv() and then the pair of inverse functions called inet_ntoa() and inet_aton(). The conv() function converts from base 10 to base 16 (or base 16 to base 10) based on the order of the arguments. The inet_ntoa() and inet_aton() functions are used to convert from an “alpha” representation of the IP address to a “numeric” representation. So the inet_ntoa() is “number to alpha” and the inet_aton() is an “alpha to number” translation.

So to summarize: the IP address comes in as 127.0.0.1 and the inet_aton() function converts that dotted notation into a number. Then the conv() function converts that number from base 10 to base 16 (hexidecimal) format, and which point it’s stored in the table. To reverse that process I have to convert from base 16 back to base 10 and then apply the reverse function inet_ntoa() to return the IP address in proper dotted notation.

I’m not sure but I think I have been told that phpBB3 does not store IP addresses using this same encoding. I’m not sure why not, as by storing the IP address encoded as hex you can use wildcards and substring operations to do some interesting queries. If the IP address is stored in dotted notation then those advantages go away. I am not sure I can come up with a good advantage to storing the IP address in dotted notation at this time. Anyone else have an idea?

As an example, in my project management system I need to be able to quickly and easily identify the parent task, if the task has any sub-tasks (child records), and which tasks are at the same level (siblings). I would like to be able to traverse the tree in either direction (up to the parent or down to the child) without using recursion. In order to do that, I need a model that is different from anything presented in the prior post.

I am going to return to the initial table design and add a few columns that will look very familiar to phpBB3 MOD authors. The new columns are called “left” and “right” which means this is the new structure for my task table:

+-----------------+-----------------------+------+-----+---------+----------------+
| Field           | Type                  | Null | Key | Default | Extra          |
+-----------------+-----------------------+------+-----+---------+----------------+
| task_id         | int(11) unsigned      |      | PRI | NULL    | auto_increment |
| task_number     | mediumint(8) unsigned |      |     | 0       |                |
| parent_task_id  | int(11) unsigned      |      | MUL | 0       |                |
| project_id      | mediumint(8) unsigned |      | MUL | 0       |                |
| task_lvl        | tinyint(2) unsigned   | YES  |     | 0       |                |
| task_left       | mediumint(8) unsigned |      |     | 0       |                |
| task_right      | mediumint(8) unsigned |      |     | 0       |                |
| task_status     | tinyint(2) unsigned   | YES  |     | 0       |                |
| task_name       | varchar(64)           |      |     |         |                |
| estimated_hours | decimal(8,2)          | YES  |     | 0.00    |                |
+-----------------+-----------------------+------+-----+---------+----------------+

There are three new columns in my task table now, and I will detail each of them next. The task_left and task_right columns are going to give me the structure that I need to do what is called a preorder tree traversal. Don’t worry, I will explain what that means shortly. With pictures. The net of it is I can recreate the task tree with a single query and therefore recursion is not required.

The task level (task_lvl in my example) can be calculated on the fly but I prefer to generate this number and store it as it saves time. The level starts at zero for a top-level task, and is incremented by one for each “step” away from the parent. So if the WBS for a task (see prior post for a definition) is 1.2.1 then that task is at level two. Count the dots and that’s the assigned level for the task. That one is easy.

Preorder Tree Traversal

I haven’t yet explained how the left and right values are assigned or used. It’s fairly simple, and the best way to describe it is to convert my task data into a tree, as shown here:

For simplicity I have only shown one branch of the task tree. This is a quite pretty picture but I can’t store the picture in the database and use it for any sort of structure. What I need is a way to convert the picture into information, and that’s easily done with what is sometimes called a “depth first” or “preorder tree traversal” process. Without going into all of the math (there is a wiki link at the end of the post for those that are interested) here is how I traverse the tree:

1. Pick up a pencil (or a crayon, in this case )
2. Put my crayon on the left side of the top node
3. Draw around the tree starting down the left side, going around each node, and back up and around until I end up on the right side of the parent node. Don’t lift the crayon, and don’t cross any lines while drawing.

Here’s what I end up with after that process:

Once I have walked the tree (following the crayon marks in the image above), I will go back over the line and each time I pass a node (task) on either the left or the right I add a number. The left side of the parent node starts at one, and I increment by one each time I pass a node. Here is the tree that has been traversed and numbered. In this case I used blue for the left side of the node, and orange for the right side of the node, for reasons which will become clear later in the post.

Now that I have made a mess of things, what do I do next? It turns out that the left and right values are very interesting numbers because by looking at the these values for a node I can determine all sorts of things with some simple observations.

First, any node where left+1 = right has to be a “leaf” or terminal node. This means with this model I can look at any node in the tree and immediately determine if it’s a leaf. That is not possible with a recursive model because I don’t know if some other node points “up” to that node as a parent without running a query first. For example, the task node 1.1 has a left value of 2 and a right value of 3, so it is a leaf. I can easily verify the rest of the leaf nodes fit this same pattern.

Second, I can determine how many children a node has with a very simple formula. (This formula works for leaf nodes as well, it’s just not needed there.) The formula is (Right - Left - 1) / 2. If I look at node 1.2.1 I see that the left value is 5, the right value is 10, and there are two child nodes. If I apply the formula I can see that (10 - 5 - 1) / 2 gives me a result of 2, which is exactly the number of children for that node. If I apply the formula to the very top node I get (16 - 1 - 1) / 2 or 7.

Conclusion

At this point I have shown how to create the tree, how the left and right values are assigned, and a few tricks that I can do with simple math. For my next post in this series I plan to start from my tree structure and show how easy it is to manage the data during operations such as adding a node, deleting a node, or moving a node. I also plan to show how easy it is to use this model to efficiently process and display any sort of hierarchical data without having to resort to recursive programming.

Related Links

• Wiki on Tree Traversal Algorithms

SQL is not a recursive language. When I write a query it’s all about relationships between rows, not about looping back through the same table. Oracle has a special construct used to traverse recursive data and it works very well, but it’s the only database that I am currently aware of to support this. Since most phpBB MOD authors will not be writing for Oracle, I will skip that concept for now.

As mentioned in the first paragraph I have recursive data in my project tracking system that I use here on the phpBB Doctor site. The design for this system is simple, but complex. The first table is the project table and it includes summary attributes for the project. These attributes include values such as when the project started, who is the project manager, a description of the project, and the status. The next table is the tasks table. A task is assigned to a project, but a task can be broken up into sub-tasks as well. There is no expected limit to the depth of the tasks. Here is a screen shot of my test project so you get an idea of what I’m talking about.

“WBS” stands for Work Breakdown Structure, and it’s a dot notation used to denote the level of detail and lineage for a project task. In this case, the first task is “1″ and the first sub-task assigned to that task is “1.1″ and the next sub-task is “1.2″. If there is a sub-sub-task then another dot is added, as in “1.1.1″ or “1.2.1″ and so on. In this way, each task has a lineage from the parent project all the way down to the most detailed level tasks. The challenge that I face is to model this data properly so that I can retrieve the lineage from top to bottom, bottom to top, or even side to side.

Task Table Design 1: Storing Parent Relationships

Here’s a typical table structure for a tasks table in this situation.

+-----------------+-----------------------+------+-----+---------+----------------+
| Field           | Type                  | Null | Key | Default | Extra          |
+-----------------+-----------------------+------+-----+---------+----------------+
| task_id         | int(11) unsigned      |      | PRI | NULL    | auto_increment |
| task_number     | mediumint(8) unsigned |      |     | 0       |                |
| parent_task_id  | int(11) unsigned      |      | MUL | 0       |                |
| project_id      | mediumint(8) unsigned |      | MUL | 0       |                |
| task_status     | tinyint(2) unsigned   | YES  |     | 0       |                |
| task_name       | varchar(64)           |      |     |         |                |
| estimated_hours | decimal(8,2)          | YES  |     | 0.00    |                |
+-----------------+-----------------------+------+-----+---------+----------------+

Notice that there is a task_id column which will be assigned by the database as the new record is inserted (that’s what the auto_increment attribute does). The task is also assigned a parent_task_id value, so that the task is related to (joined to) the parent. This allows me to walk through the task list to show task parents with their children. But in order to do that, I need a recursive language (php works) and I need to run a query for each level of the task depth. That’s not very efficient. But if I don’t do recursion, then it’s hard to put the information together correctly. Here is some of the raw data from my sample project shown in the screen shot earlier:

+---------+----------------+-----------------------------------------------+
| task_id | parent_task_id | task_name                                     |
+---------+----------------+-----------------------------------------------+
|      43 |             40 | Child Task 2 of Parent Task 1                 |
|      42 |             40 | Child Task 1 of Parent 1                      |
|      41 |              0 | Parent Task 2                                 |
|      40 |              0 | Parent Task 1                                 |
|      44 |             41 | Child Task 1 of Parent Task 2                 |
|      45 |             41 | Child Task 2 of Parent Task 2                 |
|      46 |             43 | Child task 1 of child task 1 of parent task 1 |
|      47 |             43 | Child task 2 of child task 1 of parent task 1 |
|      48 |             43 | Child task 3 of child task 2 of parent task 1 |
|      49 |             48 | Child task at level 4                         |
|      50 |             48 | Child Task 2 at level 4 After Edit            |
|      51 |             46 | Another child                                 |
|      52 |             46 | Another child                                 |
+---------+----------------+-----------------------------------------------+

If I look carefully, I can see where a parent_task_id references a task_id from another row in the output. There are two tasks where the parent_task_id is zero, which means they are the top task or starting point for the hierarchy. For example, the task that is assigned task_id 46 has a parent task of 43, and that task (43) has a parent task of 40, and that task (40) has a parent task of 0. By examing the recursive relationship of one task to another I can go up (or down) the task list.

But each time I get a row, I have to run another query based on the result of the last query in order to determine the parent. If I were trying to determine the child, I would go in the other direction, but the results would be the same. Surely there is a better way to address this.

Task Table Design 2: Separate Tables For Task Levels

I could store each different task level in a different table. If I have five levels of tasks, I could have task_01, task_02, and so on up to task_05. By doing this I am no longer doing a recursive join as each table is separate. Recursion is required because all of the data is stored in one place. Storing each level in a different table fixes that.

But don’t do this, just don’t. It’s a horrible database design. Why?

Consider what happens if some tasks have fewer than five levels of detail. In order to avoid dropping rows, each query has to be an outer join. Outer joins are typically less efficient than inner joins so I try to avoid them. And this is not the worst issue. What if some projects end up with tasks at six levels deep? Or seven? Each time I add a new level to the task structure, I have to add a new table to my database. Each time I add a new table to my database, I have to adjust my application code. The application should drive the data, not the other way around. The reality is, unless you know that you will forever have a specific depth to your recursive data, you should never consider this option.

Task Table Design 3: Ancestor – Parent – Child Model

One of the challenges of working with recursive data is finding out where a row belongs. If you have to follow the relationship up through 20 levels of data it’s going to take a while to figure out who the ultimate parent is. To solve this, I can store the root node or “ancestor” on each row. That would make the data look like this:

+-------------+---------+----------------+-----------------------------------------------+
| ancestor_id | task_id | parent_task_id | task_name                                     |
+-------------+---------+----------------+-----------------------------------------------+
|          40 |      43 |             40 | Child Task 2 of Parent Task 1                 |
|          40 |      42 |             40 | Child Task 1 of Parent 1                      |
|          41 |      41 |              0 | Parent Task 2                                 |
|          40 |      40 |              0 | Parent Task 1                                 |
|          41 |      44 |             41 | Child Task 1 of Parent Task 2                 |
|          41 |      45 |             41 | Child Task 2 of Parent Task 2                 |
|          40 |      46 |             43 | Child task 1 of child task 1 of parent task 1 |
|          40 |      47 |             43 | Child task 2 of child task 1 of parent task 1 |
|          40 |      48 |             43 | Child task 3 of child task 2 of parent task 1 |
...

In this way I no longer have to follow the relationships to find out where the starting point (the ancestor) is because the application is responsible for maintaining this data on the row itself. With this I can easily identify all of the child tasks that belong to parent task 40 (including the parent task of 40 itself) by running a query to return all rows with that ancestor_id.

This solves part of the problem (going up the path) but not all of it. For example, even if I know the ancestor, and I know all of the child tasks that belong to that ancestor, I don’t know the topology of those tasks. The topology is the layout of the tasks, showing the entire structure from top to bottom. If I refer back to the screenshot towards the top of the screen I can see how the tasks are indented based on their level. The indentation also shows which tasks are siblings (occur at the same level) as well as the parent and child status. So I’m still not quite ready to accept this design for my application.

For what it is worth, I did use this design for a real-world project for a major manufacturing company, and they are very happy with the results. I created a report from their SAP system that allowed them to determine in 30 minutes or less the answer to questions that took SAP over eight hours to generate. Cool stuff.

Task Table Design: ???

There is a better way to store recursive data, and I’m going to detail that in the next post in this series. Here is a hint: If you are working with phpBB3 then you are already using this solution. Details next time.

Related Links

• Wiki on WBS

I will not install code for clients that was written by someone else without reviewing it first. There is a lot of bad code out there. There is also some very good code, don’t get me wrong about that. But many excellent MOD authors just don’t know databases as well as they would like. So the queries that they write may be functional, but not optimal. At this point in my involvement with phpBB I write most of my own MODs, so most of the time this review is for a private client who wants me to install something that I don’t already have or have no interest in writing.

At one point after I essentially rewrote a MOD for one of my clients they suggested that I offer this service via the Doctor Board. Hmm. Well. It’s an interesting idea. It wasn’t in my plans for this board when I started (and still isn’t) and I would much rather get my desired services completed and released first, but yeah, I could do that.

Probably in about 2012.

Maybe.

The reality is I won’t offer this service, for a variety of reasons. It’s a lot of work, and the level of effort goes up substantially on complex MODs. So instead, here are a few tips that I will offer. If you are a MOD author these are the things that send up red flags to me when I read code.

Queries inside a loop

There are good reasons to put a query inside a loop. There are many more bad reasons to put a query inside a loop, especially authorization checks. Each auth check in phpbb makes two queries. I recently discovered that one of the first custom pages I created for myself was doing this. When I fixed it the number of queries per page dropped from well over 100 to 10. That’s 90+ queries that got eliminated. My server is running 30+ queries a second, so dropping 90 queries from a single page view is a substantial improvement. My current project (one of about a dozen #1 priority projects) is tuning my biggest board by going over every page specifically looking for queries that can be moved out of a loop, combined with another query, or eliminated altogether.

Some examples of MODs that I have seen that use extra queries inside a loop include some of the fancy “color group” MODs, and MODs that display a tool-tip (title attribute) that contains the text of the post. In both of these cases the MODs I rewrote the MOD that I was asked to use and they became much more efficient as a result.

Queries inside a loop can be ugly.

More than one query hitting the same tables

When I installed the Attachment MOD I noticed that the attachment icon was not visible on the search results. In order to know whether to show the icon or not, I had to do two things. First I had to know if the topic had an attachment or not, so I included the topic_attachment column to one of the standard queries. Then I had to know if the user viewing the search results was authorized to view downloads or not, so I created a new query to get the forum information. This was only one query, and it was not inside a loop.

During my review (mentioned above) I noticed that I was already hitting the phpbb_forums table in an earlier query in order to get the forum name for the search results output. So I dropped this extra query and simply added the auth_download column to the existing query. Now my search page uses one less query.

I think MOD authors end up with this situation because they don’t want to alter the existing query so they write their own instead. And in some cases having two smaller queries can be much more efficient than one really big one. But in general if you are hitting the table already, adding one more field to the select list does not have an impact on performance. Adding a second query will.

Don’t recalculate something already stored

This example comes from something that I did for a client. This was the “Top Posters on Index” MOD, and what it was supposed to do was display the top “N” members of the board ranked by post count. You may know that the user_posts field does not get updated every time a post gets deleted. In older versions of phpBB2 you could delete a post and not affect your post count. That got fixed. Pruning still does not impact the post count. What the original MOD author elected to do was recalculate the user post count every time someone viewed the index.

This is all well and good until your index starts getting hit multiple times a second. This particular client was complaining that her index took several seconds to display, and that her host was warning her about server loads. (By the way, her board is listed on big-boards.com. That means she has at least 500,000 posts and 50,000 users. This is not a small board I’m talking about.) The culprit was the Top Posters on Index MOD.

What I did was rewrite the code so that it used the user_posts field instead of recalculating the user post count every time. That should be good enough for most folks, and as a bonus, the top 10 users on the index will now match the top 10 users on the memberlist. The next thing that I did was create an index on the user_posts field so that when I do a query / order by user_posts the results come back nearly instantly. The index.php query time went from 10+ seconds to nearly instant. Oh, and the person I did the work for? I didn’t hear from her for two years after that, so I assume that the work is still holding up under an ever increasing load.

The moral of this example is that correctness (counting posts) is not necessarily as important as speed. If you can get an answer that is “good enough” and get it really really fast, that’s fine. In this case I would argue that using the user_posts field is the correct answer anyway.

Using Outer or Left Joins

Sometimes you need these. Most of the time you don’t. Outer joins are a subject for a Database Tip article that I started writing a long time ago and have not finished yet. (It will be done when… you know the rest.) Without recreating that entire article, here is the important point: Outer joins generally void index use.

Indexes exist to make your queries efficient. Earlier I mentioned creating an index on user_posts in order to speed up the Top Members on Index MOD. Without the index the code optimizations still helped. With the index the extra query added essentially zero load to the board. If you give up the index, you give up a lot of efficiency.

Avoid outer joins.

An example of an outer join is the “You posted here” MOD. There was a MOD author writing this one at phpbb.com, and he posted his SQL code. I suggested that instead of one big query he break it into two. This would allow him to avoid the outer join. He balked, at first, quoting the standard response:

Fewer queries are always better.

If you’ve been paying attention, that was what I suggested earlier in this very topic! But the point here is that you don’t always participate in every topic. Getting the “you posted here” indicator to work means you need to check and set the mark for those topics that you did post in, and leave the rest out.

This is best done with a second query.

But not a query inside the loop, no, never that.

What I do for my version of this MOD is this. First I run the general viewforum query which gets all of the topic data. I collect the topic_id values that will be shown on the page into an array. Then I run a second query that returns a count of posts that I have made per topic in the topics to be displayed on the page. I set up an array using the topic_id as the key, and the value TRUE if I have posted at least once. Now, during the viewforum output, I simply check the array for the existance of a key that matches the topic, and if I find it, I display the proper indicator.

On the search page I cheat. If you have selected one of the “canned” searches like “egosearch” then the indicator is simply turned on for every topic and I skip the extra query.

Summary

If it seemed that all of these optimization tips were related to queries, well, you are very observant. There is a reason for that. I work with databases in real life too, so that’s where I feel my expertise is. And in my experience there are a lot more “bad” MODs that can be improved by tweaking either the position or the structure of the query than by altering the actual php code.

If you have a MOD that you are thinking about, but you are wondering if it’s efficient or not, try checking it out with some of these tips in mind. If it passes, then odds are decent that it won’t kill your server. If it’s really important to you, check back in about 2016 2017 when I’ll have this service live here on the Doctor Board.

To do much of anything useful in phpBB you need data from more than one table. For example to display the index page you need information from the following tables:

• phpbb_categories
  This table contains the category name and display order
• phpbb_forums
  This table contains the forum names, the category it belongs to, the forum description, and other information (like the post_id for the last post in the forum)
• phpbb_topics
  As you might expect from the name of this table it contains the information about a topic.
• phpbb_posts
  This table contains the post information… well, most of it, anyway. It contains everything but the actual post text and things used to apply formatting (like the bbcode_uid value).
• phpbb_users
  This table is used to display the user data for the last poster in the topic.

The list goes on from there, actually. There are references to the auth_access table, the user_group table, the groups table (for figuring out who the forum moderators are) but I’ll skip those for now. The point is, each of these tables contains specific information about separate components of your board, and we want to pull everything together. That’s where joins come in.

ANSI versus “Old Style”
I’ve been working with databases for a long time. Much of my early work was done with the Oracle database but I have experience with DB2, Informix, Microsoft SQL Server, Teradata, just about all of the “big guns” in the database world. And for a long time they all did joins differently.

For example, to do an outer (optional) join in Oracle you use a syntax that looks like (+). For Microsoft you would use *=. And still other database vendors used this weird syntax with the actual words LEFT JOIN in it instead. It turns out the weird syntax was there for a reason: it’s the ANSI standard method of writing join logic. Oracle support has been spotty, at best, for phpBB2, but if you look at the current code for index.php you will still see a block of special SQL that was written just for the Oracle database.

Today I think every database that I’ve worked with (at least in the last five years) has had support for the ANSI join syntax. I tend to still use the “old” syntax, and I will talk a little bit about that. I will also talk about performance implications, and how the position of the “where” clause elements can actually change the answer to a query.

That’s a lot for one post, so we will see how far we get to start with.

Join Syntax
I will start with a very simple case of joining the categories table to the forums table and getting a list of categories and their related forums. The category table has the following structure:

+------------+-----------------------+------+-----+---------+----------------+
| Field      | Type                  | Null | Key | Default | Extra          |
+------------+-----------------------+------+-----+---------+----------------+
| cat_id     | mediumint(8) unsigned |      | PRI | NULL    | auto_increment |
| cat_title  | varchar(100)          | YES  |     | NULL    |                |
| cat_order  | mediumint(8) unsigned |      | MUL | 0       |                |
+------------+-----------------------+------+-----+---------+----------------+

The forums table has these columns (and some others that were left out for space):

+----------------------+-----------------------+------+-----+---------+-------+
| Field                | Type                  | Null | Key | Default | Extra |
+----------------------+-----------------------+------+-----+---------+-------+
| forum_id             | smallint(5) unsigned  |      | PRI | 0       |       |
| cat_id               | mediumint(8) unsigned |      | MUL | 0       |       |
| forum_name           | varchar(150)          | YES  |     | NULL    |       |
| forum_desc           | text                  | YES  |     | NULL    |       |

Notice something in common? Both tables have a cat_id field. This is a primary key (categories table) or foreign key (forums table) that I have talked about before. Suppose that I now want to pull a list of categories and their related forums, and put them in the proper order. I would use this:

SELECT	c.cat_title
,	f.forum_name
FROM	phpbb_categories c
,	phpbb_forums f
WHERE	c.cat_id = f.cat_id
ORDER BY c.cat_order, f.forum_order

This is one syntax… the “old” join syntax, where the join logic that puts the two category ID values together is in the WHERE clause. The ANSI standard syntax pulls the join logic into the FROM clause instead, and looks like this:

SELECT	c.cat_title
,	f.forum_name
FROM	phpbb_categories c
JOIN 	phpbb_forums f
	ON c.cat_id = f.cat_id
ORDER BY c.cat_order, f.forum_order

The keyword JOIN has been added, and the keyword ON identifies the join clause for the two tables. Which is easier to read? Most articles I have read suggest the second syntax is easier, although I can see arguments either way. For example in the second example it is very easy to see that the only restrictions are join restrictions; there is no “WHERE” clause and therefore no additional restrictions on the data returned by the query. When all of the joins are mixed in the WHERE clause along with other restrictions it is harder to make that determination.

Which is correct? For most databases (I should probably say all databases) the results of this query will be exactly the same. Why use one over the other? Both are fairly portable, perform equally well, and thus it would seem there’s really no difference.

But there can be. And it can be surprising. More on that in a moment.

Join Types
There are three types of joins, of which only two are typically used. The three types of joins (and their definitions) are:

• Inner Join
  This is – unless specified otherwise – the assumed type of join. An inner join looks for matching data on both sides of the join and returns row sets that fit the requirements.
• Outer Join
  Otherwise known as an “optional” join, this join will only be used if the key word LEFT or RIGHT (or possibly FULL OUTER) is included in the join clause. The word LEFT or RIGHT tells the database which side of the relationship is required with the other side becoming optional. A FULL OUTER join is optional in both directions.
• Cross or Cartesian Product Join
  This is generally an error rather than an intentional technique. A cross-product also known as a cartesian product join is really the absense of a join, and the results are generally not what you want. If you have ten forums and 100 topics and you forget to put a join clause in your query you will return every topic in every forum for a total result row count of 10 * 100 or 1,000 rows of data. A cartesian join is generally considered a bug.

JOIN or WHERE?
The main difference in ANSI versus the older style of join is to separate the join logic from the WHERE clause. The logic required to pull data out of a database can involve two steps: putting the data together, and then throwing away what you don’t want. Joins are a way to pull data together, filters or conditions are a way to throw away what you don’t want. These join filters can also appear in the FROM or the WHERE clause, and here’s where things get tricky.

I made a few very simple tables to demonstrate some join issues and differences between a join and a filter, and how it can impact the results.

First table

mysql> select * from a;
+----+-------+
| id | var1  |
+----+-------+
|  1 | Row 1 |
|  2 | Row 2 |
+----+-------+
2 rows in set (0.00 sec)

Second table

mysql> select * from b;
+----+-------+
| id | var2  |
+----+-------+
|  1 | Row A |
|  2 | Row B |
+----+-------+
2 rows in set (0.00 sec)

Associative Table

mysql> select * from a_b;
+------+------+
| a_id | b_id |
+------+------+
|    1 |    1 |
+------+------+
1 row in set (0.00 sec)

So in the sample data above I have three tables. The first table has two rows, as does the second table. The third table is an associative table that links the two together. You might notice that only one row from each table is related. In order to pull out the rows that are related I would use the following query:

select  a.*
,       b.*
from    a
join    a_b
        on a.id = a_b.a_id
join    b
        on a_b.b_id = b.id

Results

+----+-------+----+-------+
| id | var1  | id | var2  |
+----+-------+----+-------+
|  1 | Row 1 |  1 | Row A |
+----+-------+----+-------+
1 row in set (0.00 sec)

This query behaves as expected. I asked for data from all three tables, but only where a relationship exists. There is only one row that fits the qualifications, and therefore one row is presented in the result set.

Next, let’s run query that makes the join optional. We’ll require data from the left, but not the right. We will also include a condition in the WHERE clause. That query looks like this:

select  a.*
,       b.*
from    a
left join a_b
        on a.id = a_b.a_id
left join b
        on a_b.b_id = b.id
where   b.var2 = 'Row B'

The results? No rows are returned.

Empty set (0.00 sec)

The reason is that the joins are done first, then the data comparision is made against b.var2. Why do we get zero rows? Let’s move the condition (filter) from the WHERE into the FROM and rerun the query…

The SQL:

select  a.*
,         b.*
from    a
left join a_b
        on a.id = a_b.a_id
left join b
        on a_b.b_id = b.id
        and b.var2 = 'Row B'

The results:

+----+-------+------+------+
| id | var1  | id   | var2 |
+----+-------+------+------+
|  1 | Row 1 | NULL | NULL |
|  2 | Row 2 | NULL | NULL |
+----+-------+------+------+

In this case we got two rows. Hm. I said earlier that the two ways of doing joins were the same, right? so why the different results?

It’s fairly simple. When you put a filter in the FROM clause the filter is applied before (or during) the join process. If you put a filter in the WHERE clause it is done after the joins are completed. In the example above the filter is applied to table “b” and then the rows are optionally joined (via the LEFT OUTER) to table “a”. Because of the position of the filter, all of the rows from table “a” will show up. In the prior example the condition was applied after the left join was performed, therefore the rows were all eliminated. So it does, in fact, make a difference where your filters are placed.

In theory, a filter in the FROM clause could be more efficient, because it throws rows away before doing the join logic. Is it really more efficient? That’s a topic for another blog post.

Performance on Outer Joins
When you do an inner join the database is expecting data to exist on both sides of the relationship. Because of that it can use indexes to pull the data together. I have another post that I need to complete that discusses this issue in more detail. For now, let me just leave it as you should only use outer joins when they are absolutely necessary. In some cases, even having two separate queries is more efficient than one larger query with an outer join. Details to come later.

Summary
You probably can use either syntax that I showed in this post when writing joins. Old habits die hard, so I tend to use the older syntax (in the WHERE clause) unless I need an outer join.

You see, the user_id, topic_id, post_id, forum_id, and others are all primary key values. They are generated automatically by the system (user_id is a bit different from the rest; more on that in a bit) and used only to connect bits of information together. The user_id is meaningless to a person. But it means quite a bit if you want to connect a user to a group via an associative table as I covered in the first post in this series.

Centuries ago the world was thought to be flat. It turns out it wasn’t. Decades ago most databases were flat. As in flat text files. You had COBOL coders reading and writing ISAM files to mainframe disk packs or tape drives. In order to find anything, you had to know the record size and offset so you would know how far to read in order to retrieve your data. This worked okay for tapes which were essentially a linear data stream. It was okay for flat files stored on disk as well. But eventually someone figured out there was a better way to store data, and relational databases were born.

A relational database today is all about, well, relationships. You may have heard of an ER Diagram? The “ER” stands for Entity-Relationship. An entity is something like a forum or a topic or a user. A relationship is how two of those things get put together. For example, a user posts a topic, or the inverse, a topic is posted by a user. In order to maintain that relationship we need something unique about each user and about each topic. So now we’re ready to talk about primary keys.

A primary key is something that uniquely identifies an entity. In many cases the primary key is a single value (like user_id for a user) but it doesn’t have to be. For an example you would see a compound key if you review the phpbb_search_wordmatch table (an associative table between posts and search words). There is no single primary key for the table… instead it is the combination of post_id and word_id that is unique. For simple systems like phpBB it’s probably worth the extra effort to assign a single-column primary key for each table. For extremely complex systems it is often necessary to create multiple-column primary keys.

Ironically there is no primary key defined on (post_id, word_id) in the database for phpBB. The application doesn’t need it, strictly speaking it should be present to enforce data integrity rules.

Why is a primary key important? Why not just use the user’s username? That’s a decent question, and there are two very important reasons for not using something like a username as a primary key. First, it’s character data. Linking one block of character data to another block of character data requires a lot more bytes of traffic than linking two compact numeric values. The user_id in phpBB is defined as mediumint(8). This takes far less space to process than 25 characters of string data. An index on a numeric field is smaller and therefore much more efficient than an index on character data.

Second… and now we’re getting back to the opening paragraph from this post… a primary key should never ever change its value! Never. Never ever. By definition a primary key is used to relate that entity to other entities in your database. If I change that key value, I have to go through the entire database and change the key everywhere it exists! If you miss something, then you end up with what are called “orphan records” where you have parts of your database that don’t link up to anything else. That’s a “Bad Thing” to have. Usernames can change. A user_id should not.

So what is the solution? What if you wanted to have a sequential “user number” that did not have any gaps, and that got adjusted every time a user got delected from your database? That user number – like the username, email address, web site, and so on – should become an attribute of the user (entity). Attributes are pieces of data that we collect about an entity and store in our database. If it was really important for some reason to have a sequential user number then here’s how I would see that working:

1. Assign the next user_number on registration using a process similar to that used for user_id
2. Anytime a user is deleted execute something like the following:
   • get user number for deleted user
   • update users_table
     set user_number = user_number – 1
     where user_number > deleted_user_number

Is it worth it? Maybe. It depends on how important it is to you. Frankly I don’t see the need, or the point. But it should be done with a separate field, and never with the user_id primary key value.

I mentioned earlier in this post that the process used to generate user_id values is not the same as other primary keys within phpBB. Most of the primary keys are generated (on MySQL) via an auto-increment attribute. In other databases it would be done via a sequence. What this means is the application (phpBB) is not responsible for generating primary key values; the database will do that process automatically. This sounds good, so why isn’t the user_id handled in the same way? For whatever reason the “Anonymous” user has a negative user_id (-1). I am told that in earlier versions of MySQL (the most popular database used to run phpBB) an auto-increment attribute column must be numeric and unsigned, meaning it can only store positive numbers. Later versions of MySQL apparently don’t have this issue, as I was able to create a table with a signed integer key and insert negative values. The auto-increment, however, started with the first positive value rather than using any negative values.

We have here at the phpBBDoctor web site an online table reference. It’s a bit out of date, and that’s one of the projects that I intend to get around to on the first.* There is a link in the links section at the end of this post, and I encourage you to check it out if you haven’t seen it before. Even if it’s slightly out of date (as I write this, it could be updated soon), it is a more than adequate reference for the database design for phpBB2. (No work has started on an equivalent for phpBB3 as of yet.) One of the things that the reference shows is which columns are primary keys as well as which columns are foreign keys.

What’s a foreign key? Simply put, it’s a primary key value stored in another table in order to create the relationship. The user_id is a primary key for the phpbb_users table. It is a foreign key (poster_id) in the phpbb_posts table. That’s how we know which user entered the post. Another foreign key is topic_poster in the phpbb_topics table. That’s used to record which user started the topic. Why store the topic_poster in the topics table? The poster_id is stored on the first post in the topic, and we know which post is the first in the topic by looking up the topic_first_post_id from the topics table, so why do we need it? Speed.

For a pure database design you would not store the topic_poster on the phpbb_topics table. It’s done in phpBB for performance reasons, and is a process called Denormalization. That’s a topic for another post.

Related Links

• phpBBDoctor Table Reference
• Database Design #1: Associative Tables

* “On the first” means on the first chance I get.