A Look at SYSmark 2001

Pages: 1 2 3 4 5 6 7 8

Background and Target Market

BAPCo is a non-profit organization founded in 1991 with members that include Intel, Microsoft, Dell and even Ziff-Davis. According to the BAPCo FAQ, the organization’s charter is to “…develop and distribute a set of objective performance benchmarks based on popular computer applications and industry standard operating systems.”

Unlike the Winstone benchmarks, there is a fee for SYSmark 2001 that is used to cover administrative and development costs. Both eTesting Labs and BAPCo indicate that the applications selected are based upon popularity within various categories, and that a ‘typical user workload’ is implemented, however there appears to be some difference in the details of how this is determined.

eTesting Labs has published the results of a survey of user that helped them identify what features to include in their tests, but no details are provided by BAPCo on how they have done this. The FAQ mentions a ‘Workload Characterization’ methodology that is used to model user activity, which are then used to develop the scripts. These “…produce workloads that closely resemble actual usage patterns of the intended audience”, however there does not appear to be any definition of who the audience actually is, and my request for an interview with someone within BAPCo was met with no response, unfortunately.

The answer to question 9 in the FAQ seems to provide the most detail as to how the benchmarks are actually developed and who the target audience is. It states: “BAPCo’s method of designing benchmarks focuses on an actual PC user’s level of usage. This guideline will be followed by all participating members in benchmark development groups. Participation in benchmark development groups is open and encouraged. Each participating company receives one vote on the developed benchmark to assure objectivity.” Based upon this it seems reasonable to assume that the development of the scripts is performed by the participating companies, each of which get a vote on the final result as to whether to include it or not. It would appear, then, that the target audience is the participating members of the organization, who are hopefully trying to ensure that the benchmarks will be applicable to their current and potential customers. This is not necessarily a bad thing, as long as competing companies are members and they will be watching out for any unintentional (or intentional) bias in the tests. Unfortunately, AMD is not currently a member, which appears to be the basis for some of the charges leveled against the benchmark recently.


Pages: « Prev   1 2 3 4 5 6 7 8   Next »

Discuss (15 comments)