Compilers and Performance

Some Background

For those who may not be familiar with it, COSBI (Comprehensive Open Source Benchmark Initiative) is Van Smith’s effort to wrest control of benchmarking design from the hands of corporate interests and into the hands of the user community (please refer to Van’s Hardware Journal for more information). While this is certainly a laudable goal, it is also fraught with potential problems. Thus, when Van’s Hardware Journal published some preliminary results of the Quick CPU Test (here and here), a question that has been nagging me for quite some time was brought to the fore… just how much of an effect do different compilers have on the performance of a program, particularly across platforms?

The issue for me is that I need to have some kind of frame of reference in order to really understand something. Without a foundation, or basis, on which to compare things, I am left wondering what it all means. For example, if a manufacturer tells me that their new electric car will run for 6 hours on a single charge, I don’t really know what that means for me. Is that good? How does it compare to my gasoline powered car? I can answer the question by thinking about how much time I spend driving my car each week, and how many times I fill up with gas. This gives me a common frame of reference, and now I can feel comfortable in evaluating what ‘6 hours on a charge’ really means in terms of cost and convenience.

With my very limited experience with Windows programming and cross-platform compilers I found myself wondering how to interpret the results. Those with a lot of experience likely have a good ‘gut feel’ for such things, but asking around resulted in some differences in opinion. Some said Delphi is as good as any C compiler, and others said not. Some suggested that certain compilers would produce better performing code on specific platforms than others.

I am not comfortable with assertions that cannot be verified with some hard facts, and this seemed to be one of those situations. Many people have opinions, but nobody seemed to be able to back them up with solid evidence… so I continued to press for answers.

I did find a few references on the web on Delphi vs. C, and there seemed to be little difference in the resulting executables when the code was optimized by the programmer properly (see this page for an example) – but the question of whether certain compilers favor one architecture over another was still in question. Thanks to Brian Neal (see this post in Ace’s Hardware Forum), this second question could be answered – at least to some degree.

On the website referenced in Brian’s post, there are executables generated from a variety of compilers – all solving the same problem with essentially the same algorithm. For those interested in seeing whether these are really optimally implemented, the source is provided for each – but that wasn’t really my interest. I just wanted to see if different executables would result in the same relative performance as the other compilers across all platforms.

Test Setup, Results and Conclusion

What I did was to set up three systems using as many of the same components as possible:

• Matrox G550
• Matrox 5.12.01.1200 driver
• 1024×768, 16-bit color
• IBM 75GXP 45GB HDD
• Windows 2000 SP2

The platform specific components were:

• AOpen AK77Pro(A)-133, Duron 1.2GHz (133MHz x 9), Crucial Technology PC2100 DDR, CL2
• AOpen AX34-U, PIII 1.2GHz (CuMine, 133MHz x 9), Crucial Technology PC133 SDRAM, CL2
• AOpen AX4BS, P4 1.2GHz (Willamette, 100MHz x 12), Crucial Technology PC133 SDRAM, CL2

The point of making these systems as similar as possible was not to identify whether one platform performed better than another on a per-clock basis, but to simply make things as ‘equal’ as possible. I did run a test with each executable on an Athlon XP (at 1.2GHz), and they were almost identical to the Duron results, so I didn’t bother to duplicate the entire set of results. I also might have used a DDR or DRDRAM based P4, but again, my intention here was not to compare the platforms, but the compilers. It just so happens that I had this P4 system set up and ready to perform the tests (as part of another benchmark analysis I am performing).

I then ran 6 of the executables provided using the external timer program available from the same site. The parameter I used for all runs was to generate 10,000 digits of pi. The executables I chose were from the following compilers:

• Borland C++ Build 6 (bcbcpp.exe)
• Borland Delphi 6 Update 2 (delphipi.exe)
• Metroworks Code Warrior 6 (cwc.exe and cwcpp.exe)
• Microsoft Visual Studio .NET (vsc.exe and vscpp.exe)

I ran each one 10 times to determine the accuracy of the timer, threw out the highest and lowest scores, and averaged the 8 remaining scores. Generally, these 8 scores were pretty close. I also included the complete table of results at the bottom of this article.

Here are the results