Applicability of Winstone Benchmarks
Probably the most important questions about application-based benchmarks are whether the applications chosen and the features used reflect real world usage. In the case of Business and Content Creation Winstone, much of this information has been provided by eTesting Labs in their benchmark documentation and in “Benchmark Insider” articles.
The applications used in Business Winstone are described in the Business Winstone™ 2001 FAQ as “The Business Winstone tests are “market-centered” tests. Business applications are the popular applications employed by most users every day“, which means that the top ten best selling applications are used. Few can dispute that the applications chosen for Business Winstone are among the most popular in an office environment, with the exception of Netscape Communicator. The applications are MS Word, MS Access, MS Excel, MS PowerPoint, MS FrontPage, MS Project, Lotus Notes, NicoMak WinZip, Norton AntiVirus and Netscape Communicator. Netscape Communicator was chosen over Internet Explorer due to technical issues.
The applications chosen for Content Creation Winstone were also based upon market share, but this is an area where there is much more competition than in office applications. Still, the applications chosen seem to represent popular applications (again, with the exception of Netscape Navigator). The basic concept for Content Creation Winstone is to emulate what might be involved in the creation of a typical website – arguably one of the more common business activities today. The applications include Adobe Photoshop, Adobe Premier, Macromedia Director, Macromedia Dreamweaver, Sonic Foundry Sound Forge and Netscape Navigator.
It is important to understand what features have been used in order to get a better feel for the applicability of these benchmarks to the real world. For Business Winstone 2000, a survey was run asking visitors what features of each application typically caused users to wait. The results were used to identify which features would be most likely to cause users to wait for Content Creation Winstone applications as well. The methodology used and results of this survey can be found in the following three articles
- Survey Results Define Hot Spots, Part I
- Survey Results Define Hot Spots, Part II
- Survey Results Define Hot Spots, Part III
Also of importance for those analyzing the results is to identify exactly what is being timed, since all results are essentially based upon ‘wall clock’ time. For Content Creation Winstone, this information can be obtained from a very detailed set of articles from the Benchmark Insider site, including
While these articles were written specifically about Content Creation Winstone 2000, the basic functionality does not appear to have changed with Content Creation Winstone 2001. There are also some “Benchmark Insider” articles that discuss the scripts used for Business Winstone, but these are no longer applicable to the current version of the benchmark. It would be interesting to see these articles updated.
Though the “Inside Content Creation” article indicates that the intent was to include the opening and closing of applications as part of the overall score, only two applications are actually tested this way. The first four applications are opened, and then the timer is started. During the test script, two applications are closed, and the final two are opened. My own testing with various speeds of hard drives (see Organizing Windows for Performance) shows that application start up time contributes very little to the overall score, but that data access time contributes a significant amount.
Based upon comments from users on various newsgroups and message boards on subjects such as the benefits of RAID 0, or faster hard drives, it seems that there are three distinct places where disk performance is important – boot time, application load time and application operation. Personally, I would like to see the start up time measured separately from the data access time to get a better idea of the benefits of particular setups. Until then, the use of a stopwatch may be in order…
Be the first to discuss this article!