How We Evaluate
Performance: 60 % of total scoreFeatures: 30 % of total score
Documentation: 10 % of total score
How We Measure Performance
To measure the performance of a product, we install the product in a fixed platform setup. After driver installation and configuration, we will conduct test using benchmarks we have chosen for this purpose. The result of the benchmarks will be the performance metric for that product. We place the same point for all benchmarks, in order to give a more complete view of the product's performance. Using a weighted average, an overall total performance score is then given to the product. Information regarding the test setup used will be included in the evaluation notes.What is a Benchmark?
There are many ways to define a benchmark when we're talking about computer and peripherals."A benchmark is a test, or series of test, run on a product or platform to determine the performance of that product or platform under a certain, specific load. Benchmarks may be purely synthetic, testing the product at the low level, or try to emulate real world applications or even using real world applications under certain scenarios, usually meant to represent real world situations that will produce repeatable, measurable results with small degrees of variance. Some benchmarks allow users to change the settings, so these users can evaluate different aspects of a certain product or platform. Using different settings means the product will probably give different results for each setting so only products or platform using the same settings should be compared with each other. Inevitably, there will be variances between products since they may use different methods of running the task the benchmark specifies or there are other differences (clock speed, bandwidth, instruction per cycle) between the products or platform. In this case, users must make assurances that the end data, either in binary form, image, audio, should be as close to another as possible. A note should be made in the case such assurances can not be achieved. The measurable result of any benchmark is a performance metric, an indication of what the performance of the tested product or platform under that particular test. A full, complete performance evaluation means using several benchmarks, testing them with different settings and under different loads so we can get a comprehensive 'snapshot' of what kind of performance does the product or platform offer."
In short, benchmarks are tools to measure performance, in this case either a whole computer system or a certain peripheral. Using benchmarks, we can get an idea of what performance we can expect from that system or peripheral. This way we can choose what is the best option available that will fulfill our needs. Remember that benchmarks are not evaluations, only a part of it. As you can see, we don't rely on benchmarks alone but also evaluate features and documentation. Users and readers will be wise to consider the price of a product, since price is still one of the factors influencing most people's buying decision.
But Gamers Play Games, not Benchmarks
Over the years, much attention have been given to the benchmarking scene. Developer have made both synthetic benchmarks or test emulating real-world scenarios, often working with hardware manufacturers to make sure no compatibility or performance issues exist. These tools used to exist solely as professional tools, nowadays more and more benchmarks available in the market are targeted at gaming performance and they are used to evaluate products ranging from full corporate servers to PDAs and smartphones. Professional and general users alike rely on benchmarks results to help them make their buying decisions.No disrespect to the hardworking developers of synthetic benchmarks, but their software are still not games. They may or may not reflect real world gaming performance, but that is not the issue here. We prefer to use real games for our benchmarks, so our users and readers can be sure that the performance results of a product we have tested will be the performance they can expect. Basically this means getting as close as possible to actual gameplay and of course, using the game itself.
Of course, we can't use every game in existence. Not only is that impossible, but pointless. We don't believe benchmarking every game that's available under the sun will benefit users and our readers. Instead we have choose several game titles which we feel are not only good games, but also good benchmarks. These titles have been carefully chosen to represent the various game genre available. The reason is that game developers usually use some general conventions of a genre, despite using different underlaying software or 'engines' for the games. Thus, performance of one game can often be similar to other games that share the same genre. We made this decision because based on our own experience, two games using the same engine doesn't translate to the same / similar gameplay or benchmark results. In addition to that, the titles should also be very popular with gamers in their respective genre or highly regarded by gamers and game reviewers alike.
[Previous Page]
[Go to top]
[Next Page]