Trial Number Two: A New Hope

If George Lucas can get away with starting with Episode IV then I can start with trial number two, right? Well, there is actually a reason for it. Trial number 1 was really just a POC (proof of concept) to show that the techniques I outlined last time for capturing parsing and page generation times would work. The first trial was also run before I was also given a tweaked version of the Categories Hierarchy template engine to use. Trial number one was also run on my production server which meant that the template engines were not given a “quiet system” to work with. As a result, I will skip right to the second set of numbers.

Test Configuration

Server: Dell Poweredge 800, Mandriva Linux,
Concurrent Sessions: 1
Template Engines:

• phpBB Standard
• phpBB Standard with cache
• eXtreme Styles (caching enabled)
• Categories Hierarchy (modified, caching enabled)

Total Page Refreshes Performed: 19,052

Test Results Summary

First place: eXtreme Styles (XS), total average page parse 0.0195630035 seconds
Second place: Categories Hierarchy (CH), total average page parse 0.0195651181 seconds
Third place: phpBB + cache, total average page parse 0.0245600523 seconds
Fourth place: phpBB standard, total average page parse 0.0399058900 seconds

Total average page parse was calculated simply by adding the average parse times for each of the four pages together.

A few very specific observations before I get into more detailed analysis. First, the difference between first and second place is not significant: it’s only 0.0000021146 seconds after almost 20,000 pages viewed. I probably could have simply called this a tie for first place. Second, on this trial XS “won” two pages and CH “won” the other two pages. XS was faster on index.php and memberlist.php. CH was faster on both viewforum and viewtopic, which could be called the more complex pages in the test suite.

Last observation: both XS and CH were more than twice as efficient as the standard phpBB template engine. So there is definitely room for improvement on any phpBB2 board that is still using the standard template code. Even with caching added the phpBB template engine was about 25% slower than either of the top two finishers.

So in this very simple test it’s hard to say which is the best template engine. The top two both performed twice as fast as the standard. After only one test I am not ready to pick an overall winner just yet. For the next trial I will put more load on the server and see how the various template engines hold up under stress.

Detailed Numbers

I want to go back and put the very slight difference in parsing times into perspective using more detailed numbers. Recall that the average is based on refresh times for a round-robin of pages, and that the refresh is driven by a “meta refresh” directive in overall_header.tpl. Recall also that I said that some template engines were faster on some pages than others. The template used was determined by the range of seconds on the server clock at the time the page started. This meant that I did not see the exact same number of refreshes for each page for each template engine. (It was for this reason that I made sure I had over 1,000 page refreshes for each page + template engine combination.) Why does this matter?

Perhaps the difference between the two fastest template engines could be explained by the fact that one template got to refresh more of the “easier” pages than the other. This can be checked by doing a weighted average of the total page time. The average reported earlier in this post was calculated by adding the four average times together. From a mathematical perspective that calculation is only valid if each page has exactly the same sample size. In this case, they do not. For this trial here are the numbers for page refreshes for each of the two top template engines:

With these numbers I can do a weighted average rather than adding up unweighted numbers. The formula becomes:

((index page average time * index page refreshes) +
(memberlist page average time * memberlist page refreshes) +
(viewforum page average time * viewforum page refreshes) +
(viewtopic page average time * viewtopic page refreshes) ) /
(index page refreshes + memberlist page refreshes + viewforum page refreshes + viewtopic page refreshes)

So what do the adjusted (weighted) numbers look like? Interestingly enough with this revised formula the CH template engine comes out on top, even if by only a slight margin.

Trial Number 2 (one session, little or no server load)
Page Name          TPL       Avg Parse     Views
 index.php          xs      0.0037560561    1195
 index.php          ch      0.0041811737    1190
 index.php          cache   0.0048434095    1185
 index.php          phpbb   0.0089618106    1195
 memberlist.php     xs      0.0018742958    1187
 memberlist.php     ch      0.0019982736    1194
 memberlist.php     cache   0.0028188804    1198
 memberlist.php     phpbb   0.0045720672    1183
 viewforum.php      ch      0.0054771066    1186
 viewforum.php      xs      0.0060155479    1195
 viewforum.php      cache   0.0071681616    1192
 viewforum.php      phpbb   0.0115814832    1190
 viewtopic.php      ch      0.0079085643    1191
 viewtopic.php      xs      0.0079171037    1195
 viewtopic.php      cache   0.0097296009    1188
 viewtopic.php      phpbb   0.0147905290    1188