Examining what influence benchmarks results
Graphics Card and Processor Impact: Using the Right Benchmark for Each
Unlike synthetic benchmarks which isolate each peripherals in the system when testing, game benchmarks are influenced by many factors. Processors, memory used and graphics card are the peripherals that directly influence game benchmarks. In order to know what their influence are on the benchmark results, we run the candidate game benchmarks on a fixed platform, then switching processors and using different settings. The purpose of this exercise is to determine whether or not the candidate game benchmarks are valid for evaluating processors or graphics cards. System benchmarks are benchmarks on which the results scale significantly with increases of processor's performance. On the other hand, graphics benchmarks are benchmarks on which the results scale significantly with either resolution, graphical effects or features (such as FSAA and anisotropic filtering) changes.Not All Games are Created Equal: Genre and Performance Considerations
Through testing, we find that the notion of developers designing games using normal or general conventions of the games' genre is generally true. Simple, fast paced, action games favor fast frame rates and often scales better with graphical effects, resolution and graphical features. This means they are generally good graphic cards benchmarks. Simulation games on the other hand, can go either way since they reap benefits from both graphics cards' and processors' performance increases. Sophisticated, real time strategy games often gain huge benefits only with increases in processors' performance. So by definition, strategy games are often only used as system benchmarks, while simulation and action games can be used for both.What Users Notice Most: Frame rate, Details, FSAA then Anisotropic Filtering
Since we're talking about benchmarks that reflect or represent gameplay, we must also choose the settings that users most likely use. Users with different systems will have different settings, so how do we determine the ideal benchmark settings and setup? Generally, users will define the gameplay performance through the game's frame rate, the details and effects displayed by the game and graphical features activated (such as FSAA and anisotropic filtering). Using a fixed platform, users will prefer to sacrifice graphical features rather than experiencing low frame rates or low details. They will also sacrifice details and resolution if the frame rate is still to low for the gameplay to be enjoyable.So we will conduct our test at high detail levels on a resolution of 1024 x 768, on 32 bit colors for system benchmarks. For graphics benchmarks, we will use several resolutions such as 1024 x 768, 1280 x 1024, 1600 x 1200, on 32 bit colors. We will also conduct separate test for FSAA and anisotropic filtering.
Resolution : Do People Really Play Games Above 1024 x 768, 32 bit?
The answer to that question is yes, although these gamers only cover a very small number of the population. Most gamers still play games at 1024 x 768, 32 bit color or less. There are reasons for this, usually related to monitor capabilities such as maximum resolution and refresh rates but performance is also a very significant factor. Most people don't use high end graphics card and systems, so for them choosing a high detail setting at a resolution above 1024 x 768, 32 bit color often means low frame rates and in turn interferes with gameplay. Then why test performance of cards and systems at resolutions such as 1280 x 1024 or 1600 x 1200? At these resolutions, the graphics card themselves becomes the bottle neck, thus making it a graphics benchmark. This is of course assuming the system can push very high frame rates, since nothing have changed except the resolution. Graphical benchmarks often scales better with resolution than with graphical features, so this makes benchmarking at high resolutions makes more sense than cranking up the graphical features.The Illusive Frame rate per Second: 30, 60, 120 Fps or More?
This question will always pop up, since in many ways frame rate per second does determine how much you can enjoy a game. Some people can tolerate low fps, while others can't. In fact, a scientific research concluded that the human eye can actually detect a differing single frame of image in a 200 frame per second animation. So how much fps is enough? As a general rule, frame rates below 25 or 30 fps is not desirable. Below 25 fps, our eyes will notice the lag and game interactivity suffers, thus hurting gameplay. Frame rates between 30 to 60 fps is fine and can be considered normal, so 30 fps is the minimum frame rate we want. Between 60 fps to 120 fps, the animation 'feels' really fluid and seamless, helping the gameplay because you can immerse yourself further. Above 120 fps, it doesn't really matter since most CRTs can only support maximum refresh rates of 120 Hz. In relation to significance, a product's performance can be considered a significant difference if there's at least 10 % difference between them (3 fps for 25 to 30fps, 6 fps for 30 to 60 fps, 12 fps for above 120 fps)Don't Settle for Average: The Minimum Fps Hurts the Most
Most benchmarking tools and timedemo often only produce a single result: the average frame rate. Since gameplay gets very varied, there will be times where you will experience higher or lower than average frame rates. We don't really care about the higher frame rates, since they often occur when you're looking at an empty space such as a wall or sky. However, most of us are more concerned with minimum frame rates. Since minimum frame rates can be caused by different reasons, we must be careful when evaluating a product or platform. At times, they are often caused by engine limitations or other background process not related to graphical effects and features. In these cases, the minimum frame rates are caused by system limitations, since the processor can't deliver instructions fast enough to the graphics card. But there are times when the frame rate is cause by graphical effects or features, thus making it important for our graphics evaluation.This is one area we pay much attention to when we choose our benchmarks. Since we have examined the frame rate per second for the entire timedemo or replay for the candidates game benchmarks on different settings and processors, we can determine what cause these minimum frame rates and thus make a note of it when we're making our evaluation and recommendations. If the minimum frame rates are different across different processors, they're very likely caused by system limitations. Too many of them means the benchmarks is most definitely system limited. Likewise with minimum frame rates caused by graphics card limitations.
Fps Frequency Distribution - What the Average Fps Couldn't Tell You
There are times when even using minimum, average and maximum fps don't quite show the whole picture. Sometimes real world gameplay experience tells a different story. One of the reason for this fact is frame rate distribution across the whole timedemo isn't always even, like samples within a normal distribution. In a normal distribution, samples with minimum values are often offset by samples with maximum values. That's one of the reason why we examine the frequency distribution of game benchmarks we use. With it, we can see just how long (in seconds) the benchmark spend time within a given range of frame rates.Since we're more concerned with lower frame rates than higher ones, we divide the frame rate into 5 to 10 fps ranges, starting from 25 to 125 fps. There are other instances where we're using even lower frame rates, such as 15 fps. Again, the whole idea is to provide you a more complete picture of what's happening rather than just showing the results such as minimum, average and maximum fps.
[Previous Page]
[Go to top]
[Next Page]