In Part I of this series of posts I detailed the results from my first benchmark scenario. For that scenario I only ran one session. What I mean by that is I had a single browser window open. With the meta refresh keyword set up in my overall header template I could start a refresh and walk away from the system and let it run for hours.
Given that I have four template engines to test and I want to capture statistics based on at least 1,000 page refreshes for each template engine for each of the four pages in my test case that is… hmm… carry the … multiply…
1,000 refreshes * 4 template engines * 4 pages * 2 second refresh interval
… that’s 32,000 seconds, or roughly one third of a day. I certainly don’t have the time to sit around and do nothing but watch a browser session refresh all day. So it’s nice that I can start the process and leave it alone.
What other test scenarios do I have planned?
As you might recall if you read Part I of this series the decision for which template engine was the fastest was essentially even. Both the eXtreme Styles and Categories Hierarchy template system showed very well. But most boards don’t get only one visitor at a time, they get multiple. So here are the test scenarios that I plan to run (some of them are already done, I just need to crunch the numbers before I can post.)
Scenario 2: One session running with 2 second refresh intervals (completed)
Scenario 3: 12 sessions running with 2 second refresh intervals
Scenario 4: 22 sessions running with 5 second refresh intervals
Scenario 5: One session running, 2 second refresh, updated caching code
Scenario 6: One session running, 2 second refresh, updated caching code
During the testing for these various scenarios I would frequently stop in and check the server load using the top command. During Scenario 2 the server barely showed any activity at all. If the top command showed a server load approaching 0.05 that was high. Most of the time it was either 0.00 or 0.01.
During Scenario 3 I saw a server load average of around 0.5 or 50% load.
During Scenario 4 I saw a server load average of around 80% with some spikes over 1.0. This was the entire point of Scenario 3 and 4 was to test how the template parsing engines operated under load. I regularly have between 200 and 300 concurrent sessions on my largest board. Granted many of those sessions are idle (reading the page content) whereas during my benchmarking there is never any time to read anything; it’s refresh and go to the next page. Having 22 active concurrent sessions probably simulates a load of between 250 and 300 real users, in my opinion.
Scenario 5 was a repeat of scenario 2 because I was given updated code for the caching system. It turns out the update included a bug, so those numbers will not be published.
Scenario 6 was executed last night. The reason I am rerunning the single session benchmark is because the code that I was given to test for the caching phpBB2 template engine was changed quite a bit, and the author claimed that in his testing it was faster than either eXtreme Styles or Categories Hierarchy. With the bug found during Scenario 5 it certainly was faster, mainly because it didn’t render all of the page content correctly. But the early returns for Scenario 6 are quite promising, and it does appear that it might well be the fastest.
Once that scenario is completed I will rerun the load tests as well. That will give me Scenario 7 to compare to 3, and Scenario 8 to compare to 4. The numbers for the other template engines (phpbb basic, eXtreme Styles, and Categories Hierarchy) should be essentially the same as no code changes were done. This will help confirm the testing methodology and give me over 2,000 page refreshes for each page for each template engine. That’s a large enough sample that I will be very comfortable with my numbers.
Finally, I plan to ramp up to 40 concurrent sessions and see if I can make smoke come out of my development server. That should be fun.