Background and Target Market
The first benchmarks I would like to look at are the eTesting Labs (formerly Ziff-Davis Benchmark Operation, or ZDBOp) Winstone benchmarks, specifically Business Winstone 2001 and Content Creation Winstone 2001. These are amongst the most widely used performance measurement tools, primarily because they are free, run under Windows, and are easily installed and used. ZDBOp used to also offer a subcription publication, called “Benchmark Insider”, which provided quite a few details about the design and implementation of their benchmarks, but the subscription fee of $235/yr seems to have discouraged all but the most serious benchmarking professionals. Based upon discussions with a few other publications, it is likely that only the largest actually subscribed (and the Ziff-Davis Media publications likely received theirs for free).
Earlier this year, eTesting Labs made the decision to publish most of these articles on the web and make them freely available at http://www.etestinglabs.com/bi/contents/archive.asp. Much of the information presented here can be found in these articles, though some was gathered via discussions with knowledgeable individuals.
According to a recent article in PC Magazine, called “Deep Inside PC Labs“, the testing and benchmarking operation was started in 1984 to assist product reviewers by providing procedures and benchmark scores. As is the case with many web-based publications today, most of the reviewers were freelancers. In order to keep up with the increasing complexity of hardware and software products, the benchmarking operation was spun off as the Ziff-Davis Benchmark Operation, and more recently renamed to eTesting Labs.
Today, eTesting Labs develops and makes available benchmarks for testing components, systems and networks. The Winstone ‘family’ of benchmarks is the most commonly used, and includes Business Winstone, Content Creation Winstone and BatteryMark. These are application based, system level benchmarks intended to provide an indication of what an end-user will actually see with regards to system performance. BatteryMark actually uses the Business Winstone tests with some minor modifications to test the battery life of notebooks, and will not be covered here.
What is probably not well understood by most people is who the actual target market for these benchmarks really is. The assumption might be that it is IT professionals, or even end-users, but that turns out not to be entirely accurate. A clue about the real target market is not hard to find, as the home page of eTesting Labs website says “We develop these benchmarks for Ziff-Davis Media, whose publications around the world use our benchmarks and have helped make them the industry standards.”
With the numerous claims of bias and influence through ad dollars floating around, this has caused some to look with suspicion upon these benchmarks. There are also many comments that these benchmarks have outlived their usefulness, that nobody needs more power for office type applications or that the benchmarks are just ‘old and tired’. After reading the various articles available from eTesting Labs “Benchmark Insider”, I think that these latter arguments are not all that well informed, as I will attempt to show below.
As for the claims of bias (usually towards Intel), it would be interesting to see what the response from eTesting Labs would be if the larger web-based publications offered to help fund the existing benchmarks, or a new set. Short of this, the only other way to verify the integrity of the tests is for publications to perform independent evaluations on them (not just make accusations), and publish the methodology and results for others to critique and verify. This article does not attempt to determine the validity of such claims, though future articles might.
Be the first to discuss this article!