Meaningful Benchmarks

Pages: 1 2 3 4

Benchmark Selection

Obviously, the best benchmarks would be to run each application separately, and report it separately. Unfortunately, with literally thousands of applications available on several different operating systems, this is impossible. The Winstone and BAPCo benchmarks attempt to address this by running the most popular applications (according to their market research) and reporting an ‘average’ of the individual results. Unfortunately, Winstone 99 no longer reports scores for separate application groups (it only has a single average score for all tests) and because it is the most common benchmark there is a greater likelihood that manufacturers have optimized their products and drivers for the test. Accordingly, any benchmark provided by the manufacturer (whether it be Intel, AMD, 3Dfx, etc.) should be avoided, as there is a greater chance it was chosen to ‘prove’ the superiority of their particular product, or that it has been optimized for that product.

In order to help minimize the problem of skewed test results, a variety of benchmark programs should be run and all results should be provided. Each benchmark program should also be evaluated to determine what it is measuring, and whether the tests are of current value. For example, prior to the introduction of the Pentium III, a 3DNow! optimized benchmark would be a ‘fair’ comparison of an AMD vs. non-AMD processor, however today it would only be fair if the benchmark were also SSE optimized. Of course, this takes a lot of time and effort and those who are mostly interested in attracting visitors or marketing their wares (whether it be hardware, software or ‘reviews’) do not want to invest in this. The reason for this is that the majority of users want a simple chart that they can look at which shows them what the ‘best’ component is.

The one application area where the ideal has come closest is 3D games. Several, such as Quake and Incoming, have their own built-in benchmarks to provide results in frame rates. The test in Incoming provides three numbers: highest frame rate, lowest frame rate and average frame rate. If all game benchmarks provided this level of information, we would truly have reached the ideal but the reality is that most only report average, if they have a built-in benchmark at all. In order to be truly useful, we would need a benchmark for each of the most popular game engines, but this is not the case yet either, so we are stuck with the few games that provide benchmarks along with a few synthetic benchmarks, like 3D Winstone and 3Dmark99.

As was reported on Tom’s Hardware Guide, the most important benchmark number for 3D games is the lowest frame rate, not the average. It is the frame rate during the peak action times that determines how well the player will ultimately be able to perform – especially when competing against other players. Here is a perfect example of where average frame rate can be misleading, as it is possible for two systems with exactly the same average to have vastly different ‘low’ frame rates, thereby resulting in one system having much worse actual game performance.


Pages: « Prev   1 2 3 4   Next »

Be the first to discuss this article!