Interesting Zen IPC benchmarks

By: Groo (charlie.delete@this.semiaccurate.com), November 29, 2020 9:56 am
Room: Moderated Discussions
Chester (lamchester.delete@this.gmail.com) on November 29, 2020 6:16 am wrote:
> Groo (charlie.delete@this.semiaccurate.com) on November 27, 2020 9:27 am wrote:

> > Good points. That said SPEC isn't really a benchmark for consumer systems, or wasn't meant to be.
> > The way a lot of people I talked to explained it was that no serious company running spec for the
> > right reasons, IE as a sales/comparison tool, would do OC and related tricks. Why? Because if customers
> > bought the systems and ran the bench, they would get significantly lower results and then hilarity
> > would follow. By hilarity I mean lawyers, nasty phone calls from CxOs, etc.
>
> Yeah, that's the problem with using SPEC as a benchmark for consumer systems.
>
> On the OC note, say some vendor binned chips like siliconlottery, built systems with CPUs overclocked out
> of the box, and submitted OC-ed SPEC scores. That would be fine as long as they put the OC clocks in the
> results and supported their product. And that might be ok if you're looking to purchase whole systems.
>
I've seen it done, but not by 'reputable' system vendors because they don't play in that space. From memory some have done it as a stunt to show an outlier or celebrate a milestone, but realistically it causes more problems than it solves. Why? Then customers want to buy that, especially if they are in an single thread bound workload. That means the sales staff, support, and all the rest have to explain why they can't sell the customer that part, and then it devolves into annoyance. Then you get the warranty issues, and so on. Basically not worth the pain.

> Of course it wouldn't mean much to consumers looking to select parts without such a guaranteed overclock.

Yup. Vendors run benchmarks to sell products. If they don't make said products, why spend the non-trivial sums to run it?


>
> > So for the intended market, the rules as they stand seem
> > pretty OK but probably need an update. For the consumer
> > market we are already well past the point of honesty on the
> > part of the vendors, many of them have pushed cheating
> > down into firmware and even hardware in a few credible accusations. It will only get worse.
>
> If everyone is 'cheating' by ignoring turbo time limits and/or using MCE, and consumers are likely to leave
> those options on or turn them on themselves if not enabled by default, what should reviewers do?
>
There are two schools of thought here. One is that you are doing what the system is capable of doing even though it is cheating an only applicable to a specific benchmark or scenario. I don't subscribe to this one.

The other is you expose it, call it out, and trash the vendor/product in public for cheating. This is my preferred way. That said some are unwilling to do this because they know it will hurt their income stream.

> Should they test the config customers will most likely be using?
> Or turn off those options to get a 'proper' stock result?
>
Personally I would run it both ways, point out the level of cheating, trash the vendor who cheats, and recommend the other product. If a product cheats on X and is caught, are they doing other things elsewhere that weren't? Is it a good/safe/reliable product?

> > What it comes down to is look a little deeper than the scores themselves, and
> > point it out when people do wrong. If you don't, they will keep doing it.

>
> Yeah, keep pointing it out. But most people by now know to at least wait for independent
> reviews, and to take initial manufacturer benchmarks with a grain of salt.
>
Another problem of late is companies are trying to short-circuit the independent review process. Some don't sample at all before sales, other do a teaser and pre-orders before reviews, and others just ignore the who idea of reviews. It usually blows back on them, but usually not for the short term so they 'win'. Long term they lose but that isn't in the calculus for the marketers

-Charlie
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Interesting Zen IPC benchmarksAdrian2020/11/21 08:14 AM
  Interesting Zen IPC benchmarksjuanrga2020/11/21 10:22 AM
    Interesting Zen IPC benchmarksChester2020/11/21 03:49 PM
      Interesting Zen IPC benchmarksAndrei F2020/11/22 05:08 AM
        Interesting Zen IPC benchmarksChester2020/11/22 09:33 PM
          Interesting Zen IPC benchmarksnone2020/11/23 01:59 AM
            Interesting Zen IPC benchmarksnone2020/11/23 02:01 AM
              Interesting Zen IPC benchmarksnone2020/11/23 02:01 AM
          No...David Kanter2020/11/23 08:16 AM
            No...Chester2020/11/23 03:15 PM
              No...Foo_2020/11/24 03:34 AM
                No...Chester2020/11/25 12:45 AM
                  No...Per Hesselgren2020/11/25 01:57 AM
                  No...Adrian2020/11/25 04:21 AM
          Interesting Zen IPC benchmarksDoug S2020/11/23 01:03 PM
            Interesting Zen IPC benchmarksChester2020/11/23 03:26 PM
              Programs people useFoo_2020/11/24 03:43 AM
                Programs people useJukka Larja2020/11/24 07:07 AM
                Cinebench is the new Dhrystone :) (NT)none2020/11/25 01:10 AM
              Interesting Zen IPC benchmarksjuanrga2020/11/24 08:38 AM
          Interesting Zen IPC benchmarksAndrei F2020/11/24 05:47 AM
            Interesting Zen IPC benchmarksChester2020/11/25 12:32 AM
              Questionable thoughtsbenchmark critic2020/11/25 08:41 AM
                Questionable thoughtsChester2020/11/25 03:14 PM
                Questionable thoughtsnone2020/11/26 01:14 AM
                  Links?benchmark critic2020/11/26 09:48 AM
              Interesting Zen IPC benchmarksGroo2020/11/25 01:48 PM
                Interesting Zen IPC benchmarksChester2020/11/25 04:36 PM
                  Interesting Zen IPC benchmarksGroo2020/11/26 02:46 PM
                    Interesting Zen IPC benchmarksChester2020/11/26 07:32 PM
                      Interesting Zen IPC benchmarksGroo2020/11/27 10:27 AM
                        Interesting Zen IPC benchmarksChester2020/11/29 07:16 AM
                          Interesting Zen IPC benchmarksGroo2020/11/29 09:56 AM
                            Interesting Zen IPC benchmarksChester2020/11/29 04:41 PM
                Interesting Zen IPC benchmarksRayla2020/11/26 07:46 AM
                  BenchmarksDavid Kanter2020/11/26 10:05 AM
                    Benchmarksblaine2020/11/26 01:04 PM
          Interesting Zen IPC benchmarksPer Hesselgren2020/11/24 10:11 AM
            Interesting Zen IPC benchmarksChester2020/11/25 12:42 AM
      Interesting Zen IPC benchmarksjuanrga2020/11/22 07:09 AM
        Interesting Zen IPC benchmarksChester2020/11/22 09:53 PM
          Interesting Zen IPC benchmarksjuanrga2020/11/23 01:16 PM
            Interesting Zen IPC benchmarksChester2020/11/23 02:27 PM
              Interesting Zen IPC benchmarksjuanrga2020/11/24 08:25 AM
            Interesting Zen IPC benchmarksAdrian2020/11/24 11:51 AM
              Interesting Zen IPC benchmarksjuanrga2020/11/26 04:52 AM
  The Stilt's Zen 3 IPC benchmarksDummond D. Slow2020/11/25 09:29 AM
    The Stilt's Zen 3 IPC benchmarksChester2020/11/25 04:49 PM
      The Stilt's Zen 3 IPC benchmarksDummond D. Slow2020/11/25 05:58 PM
        The Stilt's Zen 3 IPC benchmarksDoug S2020/11/26 09:19 AM
      The Stilt's Zen 3 IPC benchmarksDummond D. Slow2020/11/25 06:13 PM
        The Stilt's Zen 3 IPC benchmarksChester2020/11/26 11:24 AM
          The Stilt's Zen 3 IPC benchmarksitsmydamnation2020/11/26 03:06 PM
            The Stilt's Zen 3 IPC benchmarksChester2020/11/26 07:10 PM
            The Stilt's Zen 3 IPC benchmarksDoug S2020/11/27 04:17 PM
      The Stilt's Zen 3 IPC benchmarksjuanrga2020/11/26 05:10 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell avocado?