Nothing wrong with anads spec2006 results - see the breakdown

By: Michael S (already5chosen.delete@this.yahoo.com), May 29, 2022 1:44 am
Room: Moderated Discussions
Matt Hughes (none.delete@this.none.com) on May 28, 2022 6:32 pm wrote:
> > SPEC2017 results are available and hasn't been broken in multiple tests like SPEC2006 was
>
> Which SPEC2006 tests are broken and

I don't care about SpecFp part of 2006 suit, because IMHO it is completely unrealistic, at least in its geomean form and doesn't correlated with behaviour of any optimized FP apps, which, in turn, rarely correlated with each other. SpecFp_rate is a bit more interesting, because it decently represents at least one important aspect of system performance - practically achievable memory bandwidth.
But let's concentrate on the most interesting (IMHO) part - SpecInt (non-rate).

Here 462.libquantum is broken by several compilers.
People claimed that some other subsets also broken. I have a mixed feelings about it.
IMHO, there is an issue of unrealistic autopar, which I indeed consider as breaking the benchmarks and of the rest of optimizations, which, except for libquantum, are legal and realistic.
Even in case of unrealistic autopar the impact is not nearly of the same scale as the case of libquantum.
And, of course, autopar can't break SpecInt_rate.

> how are they broken?

462.libquantum was supposed to be limited by main memory bandwidth and sensitive to main memory latency.
Benchmark specialists found an improvement of algorithm & data layout that dramatically increased cache hit rate, effectively turning it into compute-bound. Then compilers implemented this change although it's very doubtful that required transformation is legal by rules of C Standard.

> Has some compiler cheated
> by implementing benchmark-specific optimizations for some SPEC2006 tests?

Yes.

>
> I found that 416.gamess in SPEC2006 has dead code that was optimized away by g++
> but it looks like a fix was found (gcc.gnu.org/bugzilla/show_bug.cgi?id=69368).
>
> I completely agree with your point about people cherry-picking benchmark results that confirm what they
> already believe. This is a well studied human weakness called confirmation bias. Add to that the complexity
> of modern processors and marketing weasels who don’t publish enough information for customers to know
> what a product can actually do. It’s little wonder people don’t know which benchmarks to believe.

Old suggestion "use your own workload" is still the best. And still the hardest to practice.

< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Zen 4 is really badKara2022/05/26 01:52 AM
  Zen 4 is really bad-.-2022/05/26 02:13 AM
    who is the champ right now?Michael S2022/05/26 03:36 AM
      who is the champ right now?Kara2022/05/26 04:16 AM
        Alder lake better than zen4!Kara2022/05/26 04:24 AM
          Alder lake better than zen4!Adrian2022/05/26 05:40 AM
            Alder lake better than zen4!anon22022/05/26 06:14 PM
              Alder lake better than zen4!Adrian2022/05/27 04:18 AM
                Alder lake better than zen4!Matt Hughes2022/05/28 11:15 PM
      who is the champ right now?-.-2022/05/26 05:23 AM
        who is the champ right now?Adrian2022/05/26 06:09 AM
          who is the champ right now?Andrei F2022/05/26 06:16 AM
            who is the champ right now?Adrian2022/05/26 07:10 AM
              who is the champ right now?Andrei F2022/05/26 07:37 AM
                who is the champ right now?Andrei F2022/05/26 07:59 AM
              who is the champ right now?Eric Fink2022/05/27 12:17 AM
                who is the champ right now?Adrian2022/05/27 05:41 AM
            Optimally?Anon2022/05/26 09:43 AM
              Optimally?Andrei F2022/05/26 12:10 PM
          who is the champ right now?-.-2022/05/26 06:43 PM
            who is the champ right now?Adrian2022/05/27 05:24 AM
          Nothing wrong with anads spec2006 results - see the breakdownHeikki Kultala2022/05/27 03:23 PM
            Nothing wrong with anads spec2006 results - see the breakdownDoug S2022/05/27 09:59 PM
              Nothing wrong with anads spec2006 results - see the breakdownMatt Hughes2022/05/28 06:32 PM
                Nothing wrong with anads spec2006 results - see the breakdownMichael S2022/05/29 01:44 AM
                  Nothing wrong with anads spec2006 results - see the breakdownAdrian2022/05/29 03:53 AM
    Zen 4 is really badAdrian2022/05/26 05:16 AM
      Zen 4 is really bad-.-2022/05/26 05:58 AM
        Zen 4 test conditionsAdrian2022/05/26 06:14 AM
          Zen 4 test conditionsGroo2022/05/26 06:01 PM
        Zen 4 ECCAdrian2022/05/26 06:33 AM
          Zen 4 ECC-.-2022/05/26 06:38 PM
            Zen 4 ECCAdrian2022/05/27 05:11 AM
              Zen 4 ECC-.-2022/05/27 08:30 PM
                Zen 4 ECCMichael S2022/05/28 11:19 AM
          Zen 4 ECCdrAgonear2022/05/26 11:58 PM
            Zen 4 ECCDavid Hess2022/05/30 07:29 PM
      Zen 4 is really badgoose2022/05/26 08:14 PM
        Zen 4 is really badAdrian2022/05/27 04:44 AM
          Zen 4 is really badJames2022/05/27 06:13 AM
  Zen 4 is really badJohn H2022/05/26 05:45 AM
  Zen 4 is really badDoug S2022/05/26 10:50 AM
    Zen 4 is really badDummond D. Slow2022/05/28 09:14 PM
      Zen 4 is really badDoug S2022/05/29 09:49 AM
  Please check the faxts instead of posting crap like thisHeikki Kultala2022/05/26 11:05 AM
    Zen 4 IPC2022/05/26 11:24 AM
      Zen 4 IPC and AVX-512Per Hesselgren2022/05/27 01:07 AM
        Zen 4 IPC and AVX-512Per Hesselgren2022/05/27 03:44 AM
        Zen 4 IPC and AVX-512Adrian2022/05/27 04:57 AM
          Zen 4 IPC and AVX-512anon2022/05/27 08:22 AM
  Zen 4 is really badParsnip2022/05/29 04:20 AM
    What do you mean by 3-wide desing?Heikki Kultala2022/05/29 01:05 PM
      What do you mean by 3-wide desing?Doug S2022/05/29 02:32 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell tangerine? 🍊