Tech-Hounds.com

	1024 x 768	1280 x 960	1600 x 1200		Performance Hit 1280	Performance Hit 1600
Jump in The Fire Gameplay	96.03	95.22	94.73		0.84%	1.35%
Jump in The Fire Replay	90.42	89.59	88.97		0.91%	1.60%
Greendale Replay	59.27	58.94	57.42		0.56%	3.13%
Average (entire game)	96.87	94.66	88.42		1.14%	7.33%
Average (top 5 graphical levels)	95.21	89.99	77.57		4.15%	18.38%
	1024 x 768 AA AF	1280 x 960 AA AF	1600 x 1200 AA AF
Jump in The Fire Gameplay	96.64	94.24	76.61		2.48%	20.73%
Jump in The Fire Replay	88.07	87.93	77.79		0.17%	11.68%
Greendale Replay	57.41	49.12	34.73		14.44%	39.50%

all numbers are in frame rates (except for percentages)

It may be hard to see from the graph, but the results in the table shows us that this game is system limited without AA and AF. Even with AA and AF, we only saw a change in average fps once we hit 1600 x 1200 with the Jump in the fire level. As benchmarking goes, we actually have a more informative benchmark on the developer supplied Greendale replay, simply because it scales better. So, we really didn't learn anything with this game beside the fact that replays produces lower frame rates than what you will see in actual gameplay

Conclusion:

So are timedemo and replay useless? After all, they are not able to provide results that are close to frame rate in actual gameplay. We do not think so. Unlike gameplay testing sessions runs, timedemo runs and replays are very repeatable and are very good in pointing out performance influencing factors and bottlenecks (graphics cards, processor, memory, disk writes / reads). Quake 4 really illustrate this point - most of the time you play in this game you're bottlenecked by the processor and not the graphics card with a similar setup like ours.

Also remember that gameplay testing sessions will likely have a higher variation between runs than a timedemo, simply because each run will be different in some way. Though you can minimize the effect it has on average frame rates, it would be very hard to compare per second frame rates. Since there's always the chance of higher normal variations between each runs with gameplay testing sessions, we can draw the wrong conclusions from this method of performance testing. Our experience with Serious Sam II is an example of this. The differences between results at 1024 x 768 and 1280 x 960 we got from our initial runs was actually normal variations between runs and it's only after we perform more rigorous testing do we see the entire game is very likely system limited in our setup.

Running a benchmark with a timedemo / replay is also quicker and way easier than doing a gameplay testing session. By carefully doing research and picking a level that mostly stress one component or even two components, we can get a good estimate of the average fps that's pretty close to what we can get with a real gameplay testing session. The added bonus of this is we also get a pretty good idea which level represent frame rates we will see throughout the entire game.

Our experience doing gameplay testing sessions and timedemo runs / replay teaches us that every performance test tool is not perfect. Each have their own advantages and disadvantages. Both should be used to try and 'capture' what kind of performance a hardware can offer for a particular game. However, even using both doesn't mean we will be able to capture actual gameplay frame rates in its entirety. Remember, we made a subjective decision to play the game in a particular way that will be graphically intensive and produce repeatable results. We doubt real life gaming sessions from different gamers with different styles of play will be the same as ours. However, we're pretty sure that the results portray a worst case scenario - all things being equal, you should see slightly higher frame rates in these games with a similar setup.

Another thing to note is that these results are taken from the singleplayer levels of these games. Multiplayer levels are usually 'lighter' in graphical intensity, particularly since developers have no real control (no scripted events) for each room / corridor in a multiplayer level and stick to a polygon / texture / shader budget. There are also mods to consider - some mods, like the Excessive mod for Quake fans will likely put more strain on graphics cards than the default mode of play.

Then there is a trend of more and more games are becoming 'open ended' and using free roam environments. With these new games (The Elder Scrolls: Oblivion, Need for Speed: Most Wanted, Grand Theft Auto etc), it will difficult if not impossible to reprensent gameplay with gameplay testing sessions or even try to cover every type of environment that's available in the game. In these cases, reviewers usually aim for a worst case scenario and conduct gameplay testing sessions in a scene / environment that's more graphically intensive than others. Of course, that brings us to the first problem - results of these types of gameplay testing sessions will not reflect frame rates you will encounter in the game.

[Previous Page]
[Go to top]
[Home]