1. Full-Text Search
   I created a full-text index on the post subject and text over a year ago to see if maintaining that index would cause any performance issues. I’m happy to say that I have not seen any challenges from inserts / updates with this index in place. I’m going to be altering the search screen to allow the full syntax offered by MySQL on this type of index and hope to release that in a few months. Some of the challenges I have not yet decided how to solve are things like limiting forums – either by security or user preference – and other criteria that can be entered on the standard search screen.
2. Capture Post Revisions
   I’ve also added some code to capture post revisions. We’ve had a couple of folks that come back to our board and edit their post, removing all of the text and leaving only something like “…” instead. This destroys the continuity of the topic, and as a result we’re going to now track post revisions by capturing the post text history. If needed a moderator will be able to review and then restore a prior post, and ultimately lock that post from further editing. As with the full text search I have done fairly extensive testing on how this is implemented in order to ensure that performance does not suffer, and I’ll have a few blog posts about that process. This MOD is completed and I expect to roll it out onto the main board in a few weeks. (FWIW, I first talked about this post several years ago, and am just now finally getting it completed.)
3. Moderator Posts
   I’ve added a new field to the post table that allows a moderator to designate whether it’s a moderator post or a user post. For example, moderators can certainly participate in a normal board conversation as a regular person. But they may also add posts in their role as a moderator. This new feature will format those posts differently so they stand out, will automatically remove the “personal” aspects of a post such as signatures, and does not increment a moderator post count for this type of post. It is intended to be a way for moderators to be able to separate out their moderator posts from their board participation posts. This MOD is also completed and expected to be released shortly.
4. Including External Content
   I’ve added some cron jobs that parse RSS feeds from several blogs owned by board members. Their blog posts are automatically set up as part of their signature (as “Latest Blog Posts”) and updated once an hour. For bloggers that our community wants to recognize, this is a great way for them to get additional exposure without having to manually update their signature every time they publish a new blog post. This part of the MOD is already in use on our board. Only board admins can currently enter blogger information, as we want to go through a review process and certify blogs rather than allowing just anybody to link to an external site. This was done by altering the administrator user edit form and leaving the regular user profile form alone.

   As an extension to this, I’m also pulling in the content from the blog post and storing that in a hidden forum. As the blog posts are added to the forum they are obviously added to the full-text index because they’re part of the same table. I am also adding these posts to the standard phpBB2 search tables at the same time. That way if someone searches for term “X” and that’s found in an external blog post, they’ll see a link in their search results. The blog address is stored on the topic table and a different icon is used to show the user that they’re leaving our board and heading to an external site. I have all of the main work done; the last requirement is altering search.php so that it offers the ability to include / exclude external content and then react to that setting accordingly. I hope to get this completed in the next few weeks.

5. Social Media Profile Links
   I’ve added Facebook, Twitter, and LinkedIn fields to user profiles. These are displayed along with the other profile links, using smaller 18×18 pixel logos. I’m planning on going back and redoing the other profile links to use the same form factor but that part hasn’t been done yet. Here are the images I’ve made, using logos or other material provided by each service provider.

One thing that many of these MODs have in common is my concern for performance. We’re over 750K posts now, and still running extremely well on a server that is hosting several dozen sites, although none of them as active as our big board. Every time I touch the code performance is a primary goal. Another MOD that I’ve been planning is to port the phpBB3 posting form back to phpBB2 since it does a better job of supporting modern browsers as well as proving some additional formatting features. I haven’t even started on that yet, but I think it would be good. Now that I’ve personally switched to Chrome as my standard browser I’m noticing some interesting quirks.

So that’s what I’ve been up to for the past few months.

I discovered that one solution was clearly better than the other, but only if the proper index was created.

Disclaimer: I tested on phpBB2. The index that I created does not exist in a standard phpBB2, nor does it exist in a standard phpBB3 install, so I suspect this post applies to both.

Defining the Problem

Here is a partial quote of the original question in the phpBB3 MOD forum:

I want to select 5 random memebers from last 30 active users.

Simple enough, yes? First, get the last 30 members that have visited the board, then randomly select 5 of those. This could easily be done procedurally via php code, but it can also be done directly in the database with the correct SQL code.

Order By Random

The first suggestion given was this:

$start = 0;
$number = 30;
$sql = 'SELECT *
    FROM ' . USERS_TABLE . '
    ORDER BY user_lastvisit  DESC, RAND()'  ;
    $result = $db->sql_query_limit($sql, $number, $start);

With apologies to evil<3 who posted this there are a couple of things that I suggested changing. First is to avoid using the * to select every column in the table unless it’s absolutely needed. In this case, I made the assumption that the original poster was looking for a way to display a random five “recent visitors” somewhere on a page on the site. To do this doesn’t require every single bit of information about the user, just certain columns. If you select * then the entire row is returned. There are 73 columns in a standard phpBB3 users table, several of them varchar(255), and in my test installation all of the fields are mandatory. That means every single column has a value, even if it is just a space or other placeholder value. By setting up a specific list of columns to request in the query the amount of I/O is reduced. Less I/O should mean if all other things are equal the query will run faster because there are fewer bits and bytes to move around.

The other issue with the query as provided is that it’s not very efficient. It has two order by columns, one of which cannot possibly be indexed. (You can’t index something that doesn’t exist until the runtime of the query, so the rand() function result is impossible to tune.) Here is an abbreviated display for the explain plan for this query:

+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| select_type | type | possible_keys | key  | key_len | ref  | rows  | Extra                           |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+
| SIMPLE      | ALL  | NULL          | NULL | NULL    | NULL | 43367 | Using temporary; Using filesort |
+-------------+------+---------------+------+---------+------+-------+---------------------------------+

This shows that no indexes will be used for this query at all, which is not good. I ran this query five times in a row on my large user table and got execution times of 0.12, 0.12, 0.12, 0.13, and 0.12 seconds.

Select Last 30 Then Random 5

As you can see from the explain data above, I have 43,367 rows in my users table right now, which is fairly large. Instead of scanning the entire table, it would be much more efficient to get the last 30 visitors in one pass and then pick five members randomly from that list. I suggested this SQL to do that:

SELECT  u.user_id
,       u.username
FROM    phpbb_users u
,       (SELECT user_id
         FROM   phpbb_users v
         ORDER BY user_lastvisit desc limit 30) v
WHERE   u.user_id = v.user_id
ORDER BY rand() limit 5;

This is an interesting technique called “inline tables” as I am creating a new table on the fly by writing SQL code inside the FROM clause. Every database I have worked with supports this technique, so it should be portable. (I do not count Microsoft Access as a real database. ) What this SQL code will do is run the inline table to return a list of 30 users, then join that virtual table to the real table by user_id (which is a unique key) and randomly select five users from the joined result set.

Is it more efficient?

I ran this query five times (as I did with the other one) and got run times of 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. It seems that it’s not really that much more effective, so is there really a clear winner?

Index Key Columns

I ran this query:

show indexes from phpbb_users

Side Note: I run my SQL directly on the database using the MySQL command line, rather than phpMyAdmin. If you use the GUI interface, then you can check for keys by looking at the appropriate screen instead of doing as I documented here.

The results of the query did not show an index on user_lastvisit which is crucial to this solution. Here is the explain plan for my query without the index:

+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key     | ref       | rows  | Extra                           |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL    | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY | v.user_id |     1 |                                 |
| DERIVED     | ALL    | NULL          | NULL    | NULL      | 43367 | Using filesort                  |
+-------------+--------+---------------+---------+-----------+-------+---------------------------------+

Notice that is also scans all 43,367 user rows. That’s okay. What isn’t okay is that it does so without the benefit of an index and it also has to do some additional work since more than one table is involved. It would seem that the first query should be more efficient since it only has one explain step and the second one has three.

However, the magic of a database indexing can fix this. The driver for this entire question is the user_lastvisit column. After creating an index on this field (which is not indexed by default in either phpBB2 or phpBB3) here is the new explain plan.

+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| select_type | type   | possible_keys | key            | ref       | rows  | Extra                           |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+
| PRIMARY     | ALL    | NULL          | NULL           | NULL      |    30 | Using temporary; Using filesort |
| PRIMARY     | eq_ref | PRIMARY       | PRIMARY        | v.user_id |     1 |                                 |
| DERIVED     | index  | NULL          | user_lastvisit | NULL      | 43367 |                                 |
+-------------+--------+---------------+----------------+-----------+-------+---------------------------------+

Yup, still have to look at (or so the database optimizer thinks) all 43,367 users. But this time we do so with the benefit of an index. What is the impact?

The query without an index, remember, ran in 0.10, 0.12, 0.10, 0.10, and 0.11 seconds. After creating the index I ran the same query five times and got 0.00 seconds of execution time on every trial.

Does the index help the first query? Interestingly enough, it does not. The explain plan is identical with or without the index, and the query execution times do not improve either.

However, it gets worse. It doesn’t perform as required. My query gets the last 30 users that have logged in and then randomly selects five of those. The ORDER BY clause happens after all of the select and join process is complete, so I am ordering by RAND() on at most 30 rows. The other suggestion will pick the same five users nearly every single time. Why? The secondary sort column (the “random factor”) will only come into play if two users have exactly the same last visit time (down to the second). When you have two columns in the ORDER BY clause, the first column is the primary sort and every row returned will first be sorted by that column. If there are ties in the first column then and only then will the second column be sorted.

So the first solution suggested is the worst of both worlds: not only is it slower, it is also incorrect.

Conclusion

There may be other solutions to this. I don’t mean to present this post as the ultimate answer to this question. What I hoped to accomplish with this post was to show how two solutions that look the same are not always equivalent. Subtle differences can have a huge impact on functionality.

The second lesson is that if you’re going to be asking the same question from your database over and over you should carefully consider indexing the columns used in the WHERE or ORDER BY clauses. I did some work for someone a while back (their board is one of the top ten phpBB boards on the “big boards” site). They wanted to display the top ten posters on their index. The code as written was taking over ten seconds just to run the query and then the php code / template process still had to complete. I rewrote the code and added an index on the user_posts column and the code ran in less than a hundredth of a second.

On the other hand, too many indexes can also be a problem, so don’t go out and create an index for every single column in your database. Just the ones that truly matter. In this case, it makes a substantial difference.

One of my current clients is building out a new site. While I was online editing (using vi directly on the server, as I like to do) I found that just saving the files would sometimes take several seconds. And this is for a 2K file, not something incredibly large. So the first thing I did was run the “top” command, and here’s what I saw:

load average: 4.29, 3.06, 2.92

On some boxes, a load of 3.0 might not be bad. The “big board” that I am frequently talking about runs on the same server as the phpBBDoctor blog. If I hit 2.0 I’m really surprised. And since I have a quad-cpu box, I don’t really get worried until I get closer to 4.0. Yet the client box I’m working on is regularly over 6.0 or even 8.0 during the day. The stats posted above were captured at 2am central time, so hardly a busy time of the day. Yet still the numbers are what I would consider quite high.

I gave that information to my client and suggested they needed to talk to their host about the server load. It should not take 3+ seconds to have a file saved from vi by any stretch.

Then, it’s on to my next tool. I take the IP address for the domain (easily found with a simple “ping” command in most cases). I then use a tool like that found at www.myipneighbors.com which will tell me how many other domains also respond to that IP. Generally hosts are using virtual hosting so they don’t run out of IP addresses. By doing this a host can essentially offer unlimited domains on the same box with only one IP address. Hmmm, unlimited domains. Does that start to sound like a bad omen?

I went through this with a different client who was at a host that will remain unnamed but rhymes with why-power-web. Their very active site was on the same server that had over 500 sites, and that was the most I had personally seen. However, when I did the IP lookup on the server I was talking about above, here’s what I got: there are 737 sites sharing this IP address. 737! That’s a fairly large airplane, and way too many sites to host on a single server.

Conclusion? You get what you pay for. I am afraid that my client is going to be quite disappointed when they launch their new site on this new host.

Oh, and the host response to the first concern, about the server load being so high? It was something like, “Our boxes have multiple CPU’s so a high server load is not an issue.” They may have multiple CPU’s, but a high load is certainly an issue. We’ll have to see how long it takes them to fuss at my client because phpBB2 is “overloading” their already overloaded box.

A few days ago I posted Part III in my ongoing series where I am comparing phpBB3 to phpBB-Dave (my customized phpBB2-based board) from a feature perspective. I hopefully have made it clear that I am not claiming that my board is technically superior, as I am quite sure that it is not. The experimentation I did with the template engines several months back proved that, at the very least.

Part III included a review of the caching process for each board. A vanilla phpBB2 board does not offer any caching (outside of an early form of template caching). My version is caching quite a bit of information, as does phpBB3. I’ve reproduced that specific table here, as it will help clarify the points made later in this post.

One interesting result of Londonvasion for me was that I found out that there are team members that stop by and read my blog (at least occasionally) that don’t leave comments. DavidMJ, one of the current members of the phpBB3 development team, is apparently one of those folks. He caught me on IRC a few nights ago and offered to help explain the caching routines from phpBB3 and I was quick to take him up on his offer. Here is an excerpt from our conversation; I found it extremely enlightening.

[22:03] DavidMJ: drathbun: hey
[22:03] drathbun: greetings
[22:04] DavidMJ: I happened to catch your blog, was wondering if you wanted to know what we mean by “arbitrary” data wrt caching
[22:04] drathbun: at some point, yes, would love to
[22:04] drathbun: I hadn’t had time to investigate yet, but if you’re inclined to share, I’m listening
[22:04] DavidMJ: sure
[22:04] DavidMJ: we make a distinction between caching queries and everything else
[22:05] DavidMJ: so if we want to cache a query, all we have to do is make it so that we specify a TTL, the rest of the code is unchanged
[22:06] DavidMJ: everything else falls under “arbitrary”, we provide a nice mechanism for saying “take this array/string/object and cache it for some amount of time, I will remember because I have given it a name”
[22:06] drathbun: like smilies?
[22:06] DavidMJ: yep
[22:06] DavidMJ: so caching a query is totally unnamed while data is completely named
[22:06] drathbun: aha
[22:06] DavidMJ: it allows us to also make sure that we only cache old things wrt queries
[22:07] drathbun: so as a stupid noobish question, what exactly is cached when you say query cache? the sql or the results?
[22:07] DavidMJ: as we only will cache, and recall, something we have seen before
[22:07] DavidMJ: technically, both
[22:07] drathbun: ok
[22:07] DavidMJ: we hash the sql to be able to know that _exact_ query
[22:07] drathbun: so by caching the sql, you avoid the sql build step
[22:07] DavidMJ: we store the entire results very efficiently in 3.2
[22:07] DavidMJ: we store them quite well for 3.0
[22:09] drathbun: I do some what you might consider fairly primitive caching now…
[22:09] DavidMJ: drathbun: what do you do now?
[22:11] drathbun: what I call my “primitive” cache is just a dump to a file of a series of assignment statements
[22:11] drathbun: so things that are static, or nearly so, are included as needed rather than running queries on every page
[22:11] drathbun: I figure I’ve eliminated somewhere on the order of 500,000 queries a day from my server
[22:12] DavidMJ: drathbun: effective, but not as robust as the 3.0 mechanism
[22:12] drathbun: I’m sure it’s not
[22:12] drathbun: I have a cache-loader that checks the page being processed, and loads the cache related to those pages, also does the same for language files
[22:13] DavidMJ: ah, that is a bit strange
[22:13] drathbun: so I hope that the file I/O I added for the cache is offset by the reduced file I/O for unneeded language files
[22:13] DavidMJ: what we do is we load up caches as needed
[22:13] DavidMJ: we see if we recognize a query is in the cache, if so we load it
[22:13] DavidMJ: this way, identical queries on multiple pages are cached once, loaded once
[22:13] DavidMJ: given the same TTL, etc.
[22:14] DavidMJ: it also totally hides the caching logic
[22:14] DavidMJ: another nice trick is bypassing I/O alltogether
[22:14] drathbun: there are so many customized queries that can be run because of board permissions and so on, I never really applied any thought to caching queries because I figured it would be a lot of work for little benefit
[22:15] DavidMJ: 3.0 really does not need board permissions cached, it is all stored in a bitfield
[22:15] DavidMJ: the bitfield is stored per forum and is always easy to get to, the lookup is quite fast…
[22:16] DavidMJ: we have some issues when people do permission set up without using roles on huge boards
[22:16] drathbun: but in theory, with 20 different people online, couldn’t you have 20 different permission settings?
[22:16] drathbun: for the same forum?
[22:16] DavidMJ: yep
[22:16] DavidMJ: and it is stored with each user
[22:16] drathbun: aha
[22:16] DavidMJ: it is not cached anywhere, there is no need
[22:16] * drathbun sees a lightbulb
[22:16] DavidMJ: we grab the whole row anyway
[22:17] drathbun: right
[22:17] DavidMJ: so permissions are quite efficient
[22:17] drathbun: so you already know the permissions when you get the user data
[22:17] DavidMJ: yep
[22:17] DavidMJ: 3.0 is light years ahead of 2.0 wrt organization

So first, thanks to DavidMJ for taking the time to explain the caching routine to me. At some point I will be reading some code, but I have a much better understanding of what the phrases on the chart were intended to mean now. The way user permissions are stored sounds incredibly efficient, for one thing. The idea of being able to cache / share both the sql build output and the query results is also interesting. Clearly what is in 3.0 as far as caching is far, far above what I have implemented.

In case you missed it, “arbitrary data” is relatively static data like smilies. That’s what I thought they meant by database query caching. In that case, I do not do any query caching, only arbitrary data. So that means the new format for the feature comparison table is this:

Caching

The change doesn’t alter the way I scored this category. It just helps me understand more about how the caching routines work, and I am quite happy that DavidMJ offered to educate me.

Oh, and my favorite quote from the conversation? DavidMJ is nothing, if not bold. Here was a prediction he made about 3.2 during the conversation:

DavidMJ: 3.2 will have robust and reliability guarantees beyond anything I have seen in modern forum software

The thing is, I believe he along with the rest of the developer team can back that statement up and deliver on that prediction. I really do.

One of my clients was using an auto-group MOD to require posters to post in the “Introduction” forum on her board before they posted anywhere else. There were also certain semi-private forums that were not available (visible) until a member had reached a certain post level. Now in phpBB2 you manage permissions in two ways: by user (bad idea) or by group (much better). But there is no “event” system in phpBB2 that will automatically add (or remove) someone from a group. This is where the auto-group MODs generally come into play.

How They Work

Auto-group MODs generally operate by checking a set of rules (a query) against the user’s post count after each operation that can impact that value (the post count). If a user’s post count has increased above the set threshold they are added to the group(s) they now qualify for. If the user’s post count has decreased below a set threshold, then they are removed. Sounds simple, right? So what is the issue?

The Problem

The issue is there are lots of things that can change a user’s post count. Let’s count them.

1. A user enters a new post
2. A user deletes their own post
3. A moderator or admin deletes an individual post
4. A moderator or admin deletes an entire topic
5. The standard (automatic) pruning process removes a post
6. A board admin deletes an entire forum

Is that enough? There might be more, but that list is certainly long enough to make my point for me. The good news is that most of the options above call the same code, so it’s not like there are lots of different places to add new code. That’s not the issue. The issue is what that code has to do. Every time the user’s post count changes, I have to:

1. Get a list of groups with “auto-group” rules
2. Get a list of groups that I am already in
3. See if my new post count qualifies me for any new groups
4. See if my new post count requires me to be dropped from any current groups

Once that determination has been made, I then have to execute queries to adjust my group membership(s). All of these add overhead to something (the posting process) that should be as smooth as possible. Posting is, afterall, what we want people to do on our boards, right?

And here’s the worst part. These queries will execute each and every time I enter a post! Suppose that I need to accrue 100 posts before I am granted membership to a private group. From post one through post ninety-nine I pay the penalty of checking for new group membership. On post 100 I finally get to join, yippee! But guess what… every post from 101 and on is still going to check my group membership!

In my opinion, this is a very inefficient process, and should never be run on a busy board.

The Solution

The Forum Auth by Post Count MOD that I wrote does not require anything to be done during the posting or pruning process. If you look at the root issue (I want to grant access to a board only after a set post count is reached) the better solution is to add that information to the forum table. Afterall, in every case where you are requesting forum information (the forum name, for example) you can easily add new fields to the query. It is a simple matter to also request the “min_posts_to_view” field and compare it to my post count. How many extra queries does that take? Zero.

So in a nutshell that is the design behind the Forum Auth by Post Count (FAPC? Nah, don’t go there… ) modification that I wrote for phpBB2. Instead of overhead on every single post, I check one value that I have all the time ($userdata['user_posts']) against a value that I can easily retrieve every time I get the forum_id or forum_name from the phpbb_forums table and react accordingly. I want to talk more about the design of FAPC (see, it just doesn’t scan as an abbreviation) in my next post, because I think it uses a good approach. It isolates most of the code so phpBB2 upgrades are fairly easy, and it also makes it easy to reuse in a variety of places.

It’s not rocket science (but then again, only rocket science === rocket science) but it’s probably worth writing about. Stay tuned.

Related Links

• Release Topic at phpBB.com

The Test

I ran three trials. The first trial included Categories Hierarchy, eXtreme Styles, and the file cache code from the /contrib folder. I called it “phpbb-cache” for clarity. On the first trial I ran a single session with a two second refresh. On the second trial I ran 12 sessions with a five second refresh. On the last trial (the “load test”) I ran 24 sessions with a seven second refresh. The last one is really the most interesting, since it is extremely rare that my largest board has only a single active session.

The reason I ran these trials is I am rolling out a huge list of updates next week, and I really want to get off of eXtreme Styles. The two main reasons for that are (1) based on my testing eXtreme Styles does not scale well under load, and (2) an exploit was recently discovered in this code so I would rather drop it even though I have fixed the known exploit.

All trials were run using the same process and server that I have outlined before.

What Am I Looking For

I look at four numbers on my testing. First, I capture the specific time that it takes to parse the page using the pparse() function. But as has been pointed out elsewhere, that is not the complete picture. A template engine might parse well but do other things inefficiently. So I also track the page generation time, starting in common.php and ending in page_tail.php. That is the second number, and is probably the most effective at measuring total template throughput. Third, I track the database time (SQL time) for each page. In theory, they should be nearly identical for all template engines, since the database doesn’t change from one run to the next. Finally, I calculate the standard deviation of the total page times. What I am looking for here is consistency. The template engine with the lowest standard deviation is the most consistent, and therefore the better performer under system load.

I also try to ensure that I have a fair sample of values. In this sample I have over 500 refreshes for each template engine for each page being tested. (I am only testing pages that can be refreshed, for obvious reasons. I don’t want to have to write a script to testing posting or other input screens.)

The Results

Here are the results, sorted by page name and then by average page time.

I know, that’s a lot of numbers. However, my first observation is that phpbb-cache is the top-rated template for three of the four pages in the test. My second observation is that it is also the top-rated for three of the four standard deviation values. From these numbers I conclude that the basic file cache template engine from the contrib folder does a better job under load than either of the other two template engines that I tested.

I think that it’s ironic that the eXtreme Styles MOD benefited from the best SQL performance on the index page, yet still came in last place for total page generation time. I am not sure why the SQL times range as much as they do; I expected more consistency.

For the next step I added up the four average times, divided it by four and called it an “Expected Page” time. Here are the results:

As a reminder, these numbers were generated by having 24 concurrent sessions refreshing four pages (index, viewforum, viewtopic, and memberlist) one right after the other. This number is a general indication of how efficient each template engine is across the entire board. As you can see, the phpbb caching template did quite well.

Conclusion

Since I have a major upgrade planned for this weekend, and I want to get off of eXtreme Styles, I am going to switch over to the template_file_cache.php engine from the /contrib folder. I am using it for several reasons. First, it works very well. Second, I have not been able to get my adjusted categories hierarchy template engine to work with the attachment MOD, so I cannot consider it. The same issue is present (at least so far) with the new Speedy Template from Brainy.

Oh, and if you are wondering… the phpbb-cache template came in first place on every page on the single session test, and came in second place on every page (ch was first) on the 12 session load test.

We’ll be sticking with eXtreme Styles for now. Further details will be posted once I get everything working.

I checked and changed my apache settings. I checked my database settings. I kept checking and checking and tweaking and nothing made a difference. I finally resorted to blocking IP addresses that exhibited this behavior.

I think I may finally have figured out what was causing my problem. Believe it or not, it was some code in phpBB2.

phpBB2 is often blamed for server issues. In this case it seems that it certainly was a contributing factor. There may also be something about my apache configuration so I’m not going to place all of the blame on phpBB2. But the fix is quite simple.

Extreme Server Load

Here is a thumbnail of a screen shot from my apache server status screen, click to see the larger image:

If you click the image you’ll see that the entire screen is filled with requests from a single IP address. Each request is for a different instance of viewforum.php which – coincidentally enough – is one of the pages that puts the heaviest load on my server due to the number and type of queries involved. When I looked at my apache logs I saw that these requests come in at the rate of five or more per second! This is not a typical human behavior.

And to be honest, it is not a typical web-bot behavior either. Well-behaved bots do not overload your server with too many requests or they would soon find themselves out of a job. So what was causing this?

Offline Viewing

The first thought that I had was that someone was trying to use the “offline viewing” option from their browser. This option allows a user to specify a web site and a link depth. The browser will start with the specified page for the web site and download it. Then the browser will follow any included links up to the specified depth and also download them to the offline cache.

This was easy enough to test. I set up my browser to cache my board index page and set the link depth to three. But when I checked my apache log I didn’t see the same behavior, nor did my server load spike up. So I eliminated this as a potential source for the issue.

Web Accelerators

The next suggestion that I got was that this was an indication of a web-accelerator in use. Google offers a product that does this and Firefox has something called FasterFox. I took a look at both of these and looked for others as well. However, both of these products mentioned that they do not pre-fetch dynamic URLs. This means that a URL that includes an argument like viewforum.php?f=42 would not be pre-fetched!

Now what?

I finally resorted to blocking users by IP address anytime I saw this behavior. That was not my preferred solution, but I had to do something to keep my server from doing a death spiral. In some cases this browsing behavior would take my server down for up to 30 minutes.

In order to soften the blow somewhat I didn’t just block the users outright. Instead I sent them to this page which you will understand if you’ve seen the Seinfeld TV show.

On An Unrelated Note…

As if often the case the breakthrough presented itself while trying to solve something that was completely unrelated to this issue. I was doing some development on viewtopic, and one of my javascript scripts was not working. As part of the debugging process I did a “view source” on the page. What I saw gave me pause. Here it is (with some content removed for brevity)

<link rel="top" href="./index.php" title="title here" />
<link rel="search" href="./search.php" title="Search" />
<link rel="help" href="./faq.php" title="FAQ" />
<link rel="author" href="./memberlist.php" title="Memberlist" />
<link rel="prev" href="viewtopic.php?t=40050&view=previous" title="View previous topic" />
<link rel="next" href="viewtopic.php?t=40050&view=next" title="View next topic" />
<link rel="up" href="viewforum.php?f=99" title="..." />
<link rel="chapter forum" href="viewforum.php?f=1" title="..." />
<link rel="chapter forum" href="viewforum.php?f=63" title="..." />
<link rel="chapter forum" href="viewforum.php?f=2" title="..." />
<link rel="chapter forum" href="viewforum.php?f=61" title="..." />
<link rel="chapter forum" href="viewforum.php?f=4" title="..." />
<link rel="chapter forum" href="viewforum.php?f=40" title="..." />
<link rel="chapter forum" href="viewforum.php?f=48" title="..." />
<link rel="chapter forum" href="viewforum.php?f=78" title="..." />
<link rel="chapter forum" href="viewforum.php?f=7" title="..." />
<link rel="chapter forum" href="viewforum.php?f=96" title="..." />
<link rel="chapter forum" href="viewforum.php?f=65" title="..." />
<link rel="chapter forum" href="viewforum.php?f=8" title="..." />
<link rel="chapter forum" href="viewforum.php?f=47" title="..." />
<link rel="chapter forum" href="viewforum.php?f=9" title="..." />
<link rel="chapter forum" href="viewforum.php?f=38" title="..." />
<link rel="chapter forum" href="viewforum.php?f=10" title="..." />

As soon as I saw this I flashed back to the server load screen shot. Could this be the cause of my load? Is this why a single user would request to view every single forum on my board within seconds of each other?

Navigation Bar

It turns out that this code is a bit of a legacy at this point. It was designed to provide information used by the mozilla “navigation bar” which is a feature that has been obsolete for a very long time. Or at least I thought it was, as it seems that it has come back at some point. I found this reference to an add-on for Firefox that uses this information.

In theory this could be a good thing for webmasters. It allows them to set up chapters and titles and other information to provide more structure for their web site. But in the case of phpBB2 it doesn’t seem to work. They treat each forum as a chapter. If you have a lot of forums, then there are a lot of chapters. And if the add-on or plug-in attempts to pre-fetch the chapter information from each of those pages, well, that could cause the behavior that I have observed.

Even with that in mind, I wonder who thought it was a good idea to link to the entire member list as the “author” of the board content?

For what it is worth, these links are not included in phpBB3. They are no longer generated for my phpBB2 boards either. In the days since I have removed the code that generates the navigation bar information I have not seen a single instance of a server load over 3.0, much less 30 or even 300. Since I was not online viewing the server load for 24 hours at a time I also pulled out every instance of a call to viewforum.php from my apache logs and looked at the time stamps and other information. I did not find one occurrence of this behavior for the past 72 hours.

I am now quite confident (perhaps not 100%, but very, very close) that I have discovered the root cause of this issue. Oh, and the issue that I was researching on viewtopic that caused me to view the source? I figured that out too, just in case you were worried.

Why Me?

The next question I found myself asking myself was, “Why me?” Why hasn’t anyone else mentioned this specific issue at phpbb.com? Perhaps there are several answers. First, someone else may have reported or observed the issue without any idea of what was causing it, much like me at the beginning. So I’m not sure which search terms I would use to identify topics about this in the support forums.

Second, my board is quite large, both in terms of members and page views and – most importantly – forums. If you don’t have hundreds of people online at the same time you probably won’t see the issue. And even if you do have hundreds of people online at the same time, if you only have a few forums it’s still not going to be an issue.

I have almost 80 visible forums on this particular board at the time I write this. I generally have over 300 sessions active on my board at any given point in time and often have numbers that are much higher, as noted in this graph. This graph shows the highest number of users (total, guests, and members) online per hour over the past week on my largest board.

The darker part of the column indicates guest sessions and the lighter part indicates a member that has logged in. I track this information by capturing board statistics every ten minutes via a cron job. These number represent the maximum number of sessions for each hour for the last seven days.

Here is a similar graph showing average users instead of the maximum:

As you can see, over the course of the last seven days I have averaged over 150 users online per hour over the entire day. The board never sleeps.

Conclusion

The combination of active users + total forums + some browser plug-in that pre-fetches content spelled disaster for my server. I first started noticing the issue about eight months ago, and it might have been occuring before that. The issue became very obvious over the past several months. After stumbling on the navigation bar code and then removing it I have not seen the issue since. While not 100% conclusive, I feel like the results are enough of an indication that I wanted to share them, thus this post.

Related Links

• Remove Navigation Links

I do, however, want to complete this before I relaunch the board I am working on as I will include whichever template engine performs the best with that upgrade. So hopefully I will be back to this in the next week or so. Thanks for your patience.

So here is what I have to evaluate:

Session 1 did not include all four template engines and will not be used for the final analysis. Sessions 2 through 5 used a prototype version of the Speedy Template code and will not be used for the final analysis. Session 6 included bugged code from the Speedy Template prototype and will not be used.

Session 7, 8, 9, and 10 are where I will focus my attention at this point. What I plan to do is take each template + page combination and remove the top “n” and bottom “n” rows from the collection in order to leave exactly 1,000 page refreshes per each. By eliminating the top and bottom outlier records I hope to eliminate any advantage or disadvantage due to a brief server load. With four template engines and four pages and four data collection sessions that means I should end up with exactly 64,000 rows of data to work with. That even fits nicely within the range of my older version of Microsoft Excel.