In this article, we provide the results of our first Coding Challenge.
Though my background is primarily on the software side, I am very curious about the effects that different processor designs and features have on system performance. Being somewhat less knowledgeable about CPU architecture than many others, I have been looking for some way to see these effects in a concrete manner. Benchmarks are supposed to provide this ‘visible’ evidence, but there are probably as many different application mixes and workloads as there are users. This isn’t a problem for a corporation that has the money to spend to evaluate how their applications and workloads will perform on a given platform – but the average user, such as myself, is left wondering what these benchmarks are really measuring so they can be applied to our own usage.
That’s right, Real World Technologies has moved to a membership business model. Why? Because it seems to make sense, for a number of reasons, and for everyone. Does this mean there will be restricted access to articles? Definitely not. In fact, for the casual visitor there will be absolutely no noticable difference. All articles and the forum will continue to be publicly available.
Six years ago today (July 23, 1996), the Real World Technologies website was created – originally only as an online support site for local customers. Not the oldest technical site on the Web, but close. I had been doing local repairs and upgrades of PC systems part-time, partly as an opportunity to provide an out-of-work friend with some new job skills and partly to satisfy my own desires to get more familiar with PCs and electronics. I actually created the site on a whim for the purpose of learning HTML, using support as the excuse.
For those who may not be familiar with it, COSBI (Comprehensive Open Source Benchmark Initiative) is Van Smith’s effort to wrest control of benchmarking design from the hands of corporate interests and into the hands of the user community (please refer to Van’s Hardware Journal for more information). While this is certainly a laudable goal, it is also fraught with potential problems. Thus, when Van’s Hardware Journal published some preliminary results of the Quick CPU Test, a question that has been nagging me for quite some time was brought to the fore… just how much of an effect do different compilers have on the performance of a program, particularly across platforms?
PCMark2002 is basically a component level benchmark developed and distributed by MadOnion. Many of the algorithms used are based upon publicly available source code, however they appear to have been chosen for some specific attributes (such as working set size) and may not be the best representative examples of how a real world application will perform. According to the product description, it is designed to be a ‘unified’ benchmark to test PCs on any platform, specifically geared towards the home and office user (laptops, desktops and workstations). Last month, Extreme Tech published an article on PCMark2002, and posed the question of whether it really is a good CPU benchmark, given the complexity of today’s microprocessors. Based upon my initial review, the results are somewhat interesting, and I do think there is some promise here if the results are used (i.e., interpreted) correctly.
In the PC world, the age-old question seems to be “Should I buy a retail system, or should I build my own”. There are many reasons for building your own system, but the most common one is price. Experienced do-it-yourselfers will say you can build one cheaper than you can buy one, but this is not necessarily true. It may be true for them, because they have the experience. I’m not going to try to tell you how to build a good system, because I don’t believe this is something that can be learned from an article, or even a book.
After publishing my evaluation of the SYSmark 2001 benchmark, I received a fair amount of feedback from readers. Though most of the comments were not supportive of either the benchmark or methodology, there were several that were, and they brought up some good points, as well as pointed out a flaw in my evaluation. After corresponding with several readers, I must admit that my perception of the ‘think time’ may be incorrect, and if so I will have to modify my original conclusions. Also, I mentioned that by timing only the response time for individual actions it shouldn’t be difficult to give the results for every application as they used to in SYSmark 2000. What I had failed to consider was that in both scenarios (Office Productivity and Internet Content Creation), there are applications running in the background that will affect the overall time of each action.
The first question that will likely be asked is why I chose to evaluate this benchmark at all. What with all of the accusations and problems that have been found, it’s well known that this is a ‘broken’ benchmark – right? Well, not necessarily. Now, before passing judgement here, let me explain.
Sometime last year, charges were leveled against BAPCo accusing them of being ‘biased’ towards Intel, because their SYSmark 2001 benchmark suite seemed to heavily favor the P4. Later, it was revealed by AMD that one of the applications used in SYSmark 2001 used SSE instructions, but only if an Intel processor was installed. This caused the Athlon XP to appear to be at a disadvantage when compared to the P4. The outcry from many was that SYSmark 2001 was therefore an invalid benchmark that should not be used to measure system performance.