Testing Template Engines in phpBB2 Part I
Trial Number Two: A New Hope
If George Lucas can get away with starting with Episode IV then I can start with trial number two, right? Well, there is actually a reason for it. Trial number 1 was really just a POC (proof of concept) to show that the techniques I outlined last time for capturing parsing and page generation times would work. The first trial was also run before I was also given a tweaked version of the Categories Hierarchy template engine to use. Trial number one was also run on my production server which meant that the template engines were not given a “quiet system” to work with. As a result, I will skip right to the second set of numbers.
Test Configuration
Server: Dell Poweredge 800, Mandriva Linux,
Concurrent Sessions: 1
Template Engines:
- phpBB Standard
- phpBB Standard with cache
- eXtreme Styles (caching enabled)
- Categories Hierarchy (modified, caching enabled)
Total Page Refreshes Performed: 19,052
Test Results Summary
First place: eXtreme Styles (XS), total average page parse 0.0195630035 seconds
Second place: Categories Hierarchy (CH), total average page parse 0.0195651181 seconds
Third place: phpBB + cache, total average page parse 0.0245600523 seconds
Fourth place: phpBB standard, total average page parse 0.0399058900 seconds
Total average page parse was calculated simply by adding the average parse times for each of the four pages together.
A few very specific observations before I get into more detailed analysis. First, the difference between first and second place is not significant: it’s only 0.0000021146 seconds after almost 20,000 pages viewed. I probably could have simply called this a tie for first place. Second, on this trial XS “won” two pages and CH “won” the other two pages. XS was faster on index.php and memberlist.php. CH was faster on both viewforum and viewtopic, which could be called the more complex pages in the test suite.
Last observation: both XS and CH were more than twice as efficient as the standard phpBB template engine. So there is definitely room for improvement on any phpBB2 board that is still using the standard template code. Even with caching added the phpBB template engine was about 25% slower than either of the top two finishers.
So in this very simple test it’s hard to say which is the best template engine. The top two both performed twice as fast as the standard. After only one test I am not ready to pick an overall winner just yet. For the next trial I will put more load on the server and see how the various template engines hold up under stress.
Detailed Numbers
I want to go back and put the very slight difference in parsing times into perspective using more detailed numbers. Recall that the average is based on refresh times for a round-robin of pages, and that the refresh is driven by a “meta refresh” directive in overall_header.tpl. Recall also that I said that some template engines were faster on some pages than others. The template used was determined by the range of seconds on the server clock at the time the page started. This meant that I did not see the exact same number of refreshes for each page for each template engine. (It was for this reason that I made sure I had over 1,000 page refreshes for each page + template engine combination.) Why does this matter?
Perhaps the difference between the two fastest template engines could be explained by the fact that one template got to refresh more of the “easier” pages than the other. This can be checked by doing a weighted average of the total page time. The average reported earlier in this post was calculated by adding the four average times together. From a mathematical perspective that calculation is only valid if each page has exactly the same sample size. In this case, they do not. For this trial here are the numbers for page refreshes for each of the two top template engines:
Page | eXtreme Styles | Categories Hierarchy |
---|---|---|
index.php | 1195 | 1190 |
memberlist.php | 1187 | 1194 |
viewforum.php | 1195 | 1186 |
viewtopic.php | 1195 | 1191 |
With these numbers I can do a weighted average rather than adding up unweighted numbers. The formula becomes:
So what do the adjusted (weighted) numbers look like? Interestingly enough with this revised formula the CH template engine comes out on top, even if by only a slight margin. Categories Hierarchy (CH), total average page parse 0.0195559620 seconds My eyes get blurry trying to read and evaluate all of those leading zeros. What's next is to see if I can magnifiy this difference by putting the server under more load. I will use the exact same set of data, the exact same server, the exact same code modifications. The only thing that will be different is that instead of launching a single browser I will launch several. My hope is that this will accomplish two things. First, the difference between first and second place should become more substantial. Second, I will find out which template scales better under heavier loads. I regularly have several hundred concurrent sessions on my largest board, so this is something that I am going to be very interested to see. Details (and yes, that means more numbers) coming soon. Related Links Raw Data (Summary)((index page average time * index page refreshes) +
(memberlist page average time * memberlist page refreshes) +
(viewforum page average time * viewforum page refreshes) +
(viewtopic page average time * viewtopic page refreshes) ) /
(index page refreshes + memberlist page refreshes + viewforum page refreshes + viewtopic page refreshes)
eXtreme Styles (XS), total average page parse 0.0195832312 seconds
Difference: 0.0000272692 seconds With the simple calculation I used at first it showed XS beat CH by 0.0000021146 seconds. With these revised (and probably more correct) numbers CH wins by 0.0000272692 seconds. To be honest, it is just another tie for first place. The only difference is who shaved their nose hairs that day.
Either way it's less than a hundredth of a percentage difference between eXtreme Styles and Categories Hierarchy for this trial.
What is Next?
This data is sorted by page and then by average parse time.
Trial Number 2 (one session, little or no server load)
Page Name TPL Avg Parse Views
index.php xs 0.0037560561 1195
index.php ch 0.0041811737 1190
index.php cache 0.0048434095 1185
index.php phpbb 0.0089618106 1195
memberlist.php xs 0.0018742958 1187
memberlist.php ch 0.0019982736 1194
memberlist.php cache 0.0028188804 1198
memberlist.php phpbb 0.0045720672 1183
viewforum.php ch 0.0054771066 1186
viewforum.php xs 0.0060155479 1195
viewforum.php cache 0.0071681616 1192
viewforum.php phpbb 0.0115814832 1190
viewtopic.php ch 0.0079085643 1191
viewtopic.php xs 0.0079171037 1195
viewtopic.php cache 0.0097296009 1188
viewtopic.php phpbb 0.0147905290 1188