David Kanter (dkanter@realworldtech.com) on 3/30/11 wrote:
---------------------------
>NaN (spam@googlemail.com) on 3/30/11 wrote:
>---------------------------
>>Well, first thanks for your helpful comparison despite the >"useless" benchmarks.
>
>Yeah, don't get me started on the tests. The suite they were using is a half-assed
>pile of junk. There are things I dislike a lot about SPEC, but their benchmarks
>are a model of clarity in comparison (even without having source code available).
>It's also amusing how many of the tests were very small toy benchmarks or had horrible
>coding practices (based on commentary I had seen).
>

I believe the phoronix/openbenchmarking people intended to build a framework to facilitate creating your own benchmarks, rather than a standardized benchmark like SPEC, etc.. Thus the lack of any attempt at standardization. So irrespective of whether or not there are implementation issues this framework is probably ill suited to your needs (unless you want to create and define your own benchmark standard to run withing the framework - which I expect would be alot of work).

Also, the subset of available tests chosen for the bulldozer run doesn't help. Some apache, postgres or java benchmarks would've been more useful to us.