Do The Research
Benchmark analysis is both an art and a science. There are many things that must be taken into consideration when evaluating benchmarks. You must clearly define your objective, and choose the benchmarks that will actually test those areas you are intending to measure. If comparing components with a system level benchmark, you must ensure that all hardware be exactly the same, except for that one component that is being compared. If this cannot be achieved, you must somehow provide a method of determining exactly what effect the hardware differences have.
As an example of this, when performing the PCI tests for our article “Overclocking for Performance”, we had to use two different motherboards for testing the effect of an async vs. sync PCI clock when the system clock was set to 83MHz. You can see that we ran ‘baseline’ tests with both motherboards at 66MHz so we could see if there was any difference between the motherboards themselves. After determining what the difference was at 66MHz, we could then derive the effect of the sync vs. async PCI clock at 83MHz.
That article also ran only a few benchmarks, and reported only the average score because the objective was to compare the overall effect of running the exact same system while changing CPU settings. The effect on individual applications was considered less important than the overall performance impact.
Most everyone understands the phrase ‘There are Lies, Damn Lies and Statistics‘, but it is too often seen as only an amusing comment rather than a real warning. Someone once said ‘People believe what they want to believe, and what they read in the newspapers‘, which can now be extended to include the World Wide Web. Benchmark results are too often used improperly for the purpose of gaining bragging rights by users and manufacturers, or simply to get visitors to a site. Those wanting to truly evaluate the faster products for their purposes need to wade through the BS and extract the meaningful data, and too often there isn’t enough to come to any reasonable conclusions. By following the quidelines presented here (and in the following links), you can help reduce the proliferation of ‘benchmark pollution’.
Additional Information and Research
Andre D. Balsa wrote a series of articles for the Linux Gazette which detailed the process of benchmarking for the Linux operating system. Though specifically geared towards Linux, many of the principles he lays out apply to other operating systems as well. The first three of these articles can be found online:
- Linux Benchmarking Concepts
- GNU/Linux Benchmarking – Practical Aspects
- Linux Benchmarking Part III – Interpreting Benchmark Results
Professional Benchmark Organizations:
- Client/Server Solutions, Inc – http://www.csrad.com
- System Performance Evaluation Cooperative (SPEC) – http://www.specbench.org
- Transaction Processing Performance Council (TPC) – http://www.tpc.org
- AIM Technology – http://www.aim.com
- Ziff-Davis Benchmark Operation – http://www.zdnet.com/zdbop
Additional Information on Benchmarking:
- Benchmark Resources – http://www.benchmarkresources.com
- Rich Grace’s Benchmark Page – http://www.byteit.com/rgrace/bench.htm
- comp.benchmarks FAQ – http://sacam.oren.ortn.edu/~dave/benchmark-faq.html
- Computer Architecture Research Project – http://www.wam.umd.edu/~maymay/introduction.html
Information on Specific Benchmarks:
- Winstone/Winbench 96 information: http://www.zdnet.com/pcmag/issues/1421/pcm00154.htm
- Winstone/Winbench 97 information: http://www.zdnet.com/pcmag/pctech/content/15/21/ln1521.001.html
- Latest Release Winstone information: http://www1.zdnet.com/zdbop/winstone/winstone.html
- Latest Release Winbench information: http://www1.zdnet.com/zdbop/winbench/winbench.html
- Latest Release 3D Winbench information: http://www1.zdnet.com/zdbop/3dwinbench/3dwinbench.html
- Latest Release CD Winbench information: http://www1.zdnet.com/zdbop/cdwinbench/cdwinbench.html
- Latest Release Audio Winbench information: http://www1.zdnet.com/zdbop/auwinbench/auwinbench.html
- BAPCo benchmark information: http://www.bapco.com
Be the first to discuss this article!