mea culpa (lmbench is horribly broken)

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), March 19, 2017 6:05 pm
Room: Moderated Discussions
Linus Torvalds (torvalds.delete@this.linux-foundation.org) on March 19, 2017 4:25 pm wrote:
>
> I guess I'll have to download it for the first time in years
> and re-run it on modern hardware, just to look once more.

So doing so shows that I definitely had some rose-colored glasses wrt lmbench, and I will have to admit that lmbench output was much worse than I remembered. I guess I have a soft spot for it just because I only ever really cared about the OS microbenchmarks (the hardware I couldn't affect), which were reasonably useful back then.

In fact, even the graphs (which do indeed show stride-vs-size like I remembered) are pretty bad. They made more sense in lmbench2 - which is actually what I used almosty exclusively just to have historical data to compare against. But that doesn't actually really help, because...

But lmbench2 lacked the randomization for the memory latency chain testing, so the graphs may make sense because you can see how the stride affects things, but at the same time that ends up being useless for latency testing on modern CPU's, since it's mostly about capturing prefetching etc.

And in lmbench3, the actual raw data may have the randomized numbers, but they don't show up in the graph (even though there's a marker for it - maybe the line is just mislabeled. Regardless, the graph is not useful).

All the scripts are over a decade old and they rely on ancient unix tools (grof etc) that I didn't even have installed on my modern system, so it might be something got screwed up due to that. But it certainly isn't very pretty.

So I'm starting to have to admit that Wilco is right that lmbench is horribly broken just because it's effectively useless. Because even if I may still believe that TLB effects should show up in memory latency benchmarks, but lmbench certainly isn't really the thing to use.

(And in a perfect world, the benchmark show the pointer chasing letency both with and without TLB effects, possibly by comparing the random chain walker in a a small page allocation vs one using hugepages).

Mea culpa. Mea maxima culpa.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
ARM A73 benchmarksSymmetry2017/03/14 06:24 AM
  ARM A73 benchmarksPer Hesselgren2017/03/14 07:18 AM
    ARM A73 benchmarks-latencyPer Hesselgren2017/03/14 08:58 AM
      ARM A73 benchmarks-latencySymmetry2017/03/14 10:12 AM
        ARM A73 benchmarks-latencyPer Hesselgren2017/03/14 03:54 PM
          ARM A73 benchmarks-latencyWilco2017/03/15 01:45 AM
            ARM A73 benchmarks-latencyPer Hesselgren2017/03/15 02:57 AM
              ARM A73 benchmarks-latencyPer Hesselgren2017/03/15 03:00 AM
                ARM A73 benchmarks-latencyPer Hesselgren2017/03/15 03:01 AM
                  clickable linkMichael S2017/03/15 04:05 AM
            ARM A73 benchmarks-latencyLinus Torvalds2017/03/15 10:05 AM
              ARM A73 benchmarks-latencyIreland2017/03/15 05:02 PM
              ARM A73 benchmarks-latencyGabriele Svelto2017/03/16 03:45 AM
                ARM A73 benchmarks-latencyLinus Torvalds2017/03/16 02:01 PM
                  lmbench is horribly brokenWilco2017/03/16 04:57 PM
                    lmbench is horribly brokenLinus Torvalds2017/03/16 06:49 PM
                      lmbench is horribly brokenLinus Torvalds2017/03/17 01:10 PM
                        lmbench is horribly brokenLinus Torvalds2017/03/17 01:52 PM
                        lmbench is horribly brokenExophase2017/03/17 02:31 PM
                          lmbench is horribly brokenGabriele Svelto2017/03/17 03:20 PM
                          lmbench is horribly brokenLinus Torvalds2017/03/17 05:56 PM
                            lmbench is horribly brokenExophase2017/03/17 06:21 PM
                              lmbench is horribly brokenLinus Torvalds2017/03/17 06:43 PM
                                lmbench is horribly brokenIreland2017/03/17 07:37 PM
                                  lmbench is horribly brokenbakaneko2017/03/18 11:17 AM
                                    lmbench is horribly brokenIreland2017/03/18 12:23 PM
                                      lmbench is horribly brokenanon2017/03/18 07:35 PM
                                      lmbench is horribly brokenbakaneko2017/03/21 08:08 AM
                                        lmbench is horribly brokenIreland2017/03/21 03:14 PM
                                lmbench is horribly brokenGabriele Svelto2017/03/18 04:01 PM
                                  accessing dram RichardC2017/03/18 06:33 PM
                                lmbench is horribly brokenExophase2017/03/18 04:26 PM
                                  lmbench is horribly brokenWilco2017/03/18 05:40 PM
                                    benchmarking reality?Anon2017/03/19 02:29 PM
                                    lmbench is horribly brokenLinus Torvalds2017/03/19 04:25 PM
                                      mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/19 06:05 PM
                                        mea culpa (lmbench is horribly broken)Bill Broadley2017/03/21 01:41 AM
                                          mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/21 09:01 AM
                                            mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/21 11:14 AM
                                            mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/21 05:03 PM
                                              mea culpa (lmbench is horribly broken)Etienne2017/03/22 04:37 AM
                                              mea culpa (lmbench is horribly broken)Tim McCaffrey2017/03/22 08:54 AM
                                                mea culpa (lmbench is horribly broken)Tim McCaffrey2017/03/22 09:34 AM
                                                mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/22 10:35 AM
                                                  mea culpa (lmbench is horribly broken)Ireland2017/03/22 12:11 PM
                                                    mea culpa (lmbench is horribly broken)Ireland2017/03/22 12:26 PM
                                                    mea culpa (lmbench is horribly broken)rwessel2017/03/22 03:03 PM
                                                      mea culpa (lmbench is horribly broken)Ireland2017/03/22 03:35 PM
                                                  mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/22 01:35 PM
                                                    mea culpa (lmbench is horribly broken)Gabriele Svelto2017/03/23 08:05 AM
                                                      mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/23 10:43 AM
                                                        mea culpa (lmbench is horribly broken)Gabriele Svelto2017/03/23 01:56 PM
                                                          mea culpa (lmbench is horribly broken)Ireland2017/03/23 02:36 PM
                                                  mea culpa (lmbench is horribly broken)Travis2017/03/22 01:38 PM
                                              mea culpa (lmbench is horribly broken)anon2017/03/22 07:22 PM
                                                mea culpa (lmbench is horribly broken)Travis2017/03/22 08:57 PM
                                                  mea culpa (lmbench is horribly broken)anon2017/03/23 12:44 AM
                                                    mea culpa (lmbench is horribly broken)Michael S2017/03/23 05:59 PM
                                                      mea culpa (lmbench is horribly broken)Travis2017/03/23 09:03 PM
                                                    power8 numbersoctoploid2017/03/24 11:47 PM
                                                      power8 numbers stride=128octoploid2017/03/25 04:36 AM
                                                        power8 numbers stride=128Linus Torvalds2017/03/25 10:50 AM
                                                          power8 numbers stride=128Gabriele Svelto2017/03/25 11:27 PM
                                              mea culpa (lmbench is horribly broken)anon2017/03/23 01:14 AM
                                                mea culpa (lmbench is horribly broken)Linus Torvalds2017/03/23 11:22 AM
                                                  Thank you. Associativity misses explain it.anon2017/03/23 10:48 PM
                                                    Thank you. Associativity misses explain it.Linus Torvalds2017/03/24 01:26 PM
                                                      Thank you. Associativity misses explain it.Travis2017/03/24 10:01 PM
                                                        thanks should read "but if it is any TYPE of mix" (NT)Travis2017/03/24 10:02 PM
                                                        Thank you. Associativity misses explain it.Linus Torvalds2017/03/25 12:10 PM
                                                          Thank you. Associativity misses explain it.Travis2017/03/25 04:08 PM
                                                            Thank you. Associativity misses explain it.Linus Torvalds2017/03/26 10:27 AM
                                  lmbench is horribly brokenLinus Torvalds2017/03/19 03:51 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?