Applicability of the SYSmark 2001 Benchmark
Unlike eTesting Labs, there is little information regarding the selection process for applications and the workload characteristics. It does appear from the FAQs that some data is collected that identifies the retail sales of various office and Internet content creation applications, which is then used to select the specific components of the suite. From this it should be safe to assume that the applications are representative of a typical work PC. Whether the workload is representative or not is another matter, but in looking at the script overviews in the White Paper, it would appear to not be an issue. As mentioned at the beginning of this article, there have been accusations of bias due to the selection of Windows Media Encoder 7.0, but I believe I have answered that question.
The main concern I have with this benchmark is whether the measurement methodology really provides an indication of what an end user will experience. As indicated above, the methodology used essentially eliminates swapping and other OS activities from the measurements with the ‘think time’ delays. Can you really call this a ‘system level’ benchmark if you eliminate that activity from the measurements? I would contend that it may measure the performance of the specific applications used, but it does not measure system performance. Experienced users know that adding more memory will speed up overall system performance in many cases, so it will be interesting to see the effects of this on the SYSmark 2001 benchmark.
Discuss (15 comments)