lmbench is horribly broken

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), March 19, 2017 4:25 pm
Room: Moderated Discussions
Wilco (Wilco.Dijkstra.delete@this.ntlworld.com) on March 18, 2017 5:40 pm wrote:
>
> Actually once you start missing the LLC you end up doing 2 DRAM accesses just to get the TLB entry plus a third
> for the actual access.

Bullshit, Wilco.

Why are you making up these things? Just admit that you're full of crap already.

The page tables cache so well that they'll fit in caches even for large areas. In particular, you'll see a lot more misses for the actual data access itself than you will for the TLB, because the page table accesses are denser by a factor of 512 (assuming a 4kB page and a 8 byte paga table entry).

The lmbench memory testers recommend using an area that is at least 4x the size of your last-level caches. Say you pick 8x or 16x just to add some extra padding. Even when you are missing most of the time in the last-level D$, you will be hitting quite a lot in the page table caches, because of that much higher density.

> Basically it doesn't matter how advanced your TLB is, you quickly end up getting the worst case of 2-3
> DRAM accesses per load.

No. You're just making shit up now.

Yes, the absolute theoretical worst case is that you'll take a cache miss on every single page table level, in addition to the cache miss on the actual data.

But see above - that's not what lmbench will be measuring, so you are bringing up a total red herring. That theoretical absolute worst case has nothing to do with the subject line (which is your completely bogus assertion that lmbench is horribly broken).

lmbench isn't perfect, it's old, it's not maintained afaik, it tests a lot of things that aren't necessarily very relevant. And it's fundamentally a microbenchmark, which is pretty debatable to begin with. It's certainly no "specint gcc" kind of measure of any real life performance, it's just a collection or random OS/hw microbenchmarks.

There are better things for almost all of what it tests. But that said, lmbench numbers are generally more relevant than you give them credit for, and in particular, they are almost certainly a lot more real and relevant than many of the GB4 numbers are - including the memory latency ones.

And once more, I will state that the lmbench "summary" numbers are not necessarily the greatest, and they aren't even comparable to each other if only because many of the summary numbers depend on what parameters you gave to lmbench when you set it up (eg the whole "how big a memory area do you want to use for testing" etc).

So you simply don't see a lot of the data that lmbench actually gathers in those summaries. For some of the tests the summary is pretty much all you would ever want to see, but for the memory latency benchmarks the real meat really is in the full data graphs.

And it does a fairly good job at that, and can show interesting data.

In particular, the memory latency numbers will *not* be showing the cost of 2-3 extra memory cycles for each access. Not with a reasonable TLB. That's just bullshit, Wilco. I ran lmbench for years (back when it _was_ actively developed), and it gave quite realistic memory latency numbers.

I guess I'll have to download it for the first time in years and re-run it on modern hardware, just to look once more.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
ARM A73 benchmarksSymmetry03/14/17 06:24 AM
  ARM A73 benchmarksPer Hesselgren03/14/17 07:18 AM
    ARM A73 benchmarks-latencyPer Hesselgren03/14/17 08:58 AM
      ARM A73 benchmarks-latencySymmetry03/14/17 10:12 AM
        ARM A73 benchmarks-latencyPer Hesselgren03/14/17 03:54 PM
          ARM A73 benchmarks-latencyWilco03/15/17 01:45 AM
            ARM A73 benchmarks-latencyPer Hesselgren03/15/17 02:57 AM
              ARM A73 benchmarks-latencyPer Hesselgren03/15/17 03:00 AM
                ARM A73 benchmarks-latencyPer Hesselgren03/15/17 03:01 AM
                  clickable linkMichael S03/15/17 04:05 AM
            ARM A73 benchmarks-latencyLinus Torvalds03/15/17 10:05 AM
              ARM A73 benchmarks-latencyIreland03/15/17 05:02 PM
              ARM A73 benchmarks-latencyGabriele Svelto03/16/17 03:45 AM
                ARM A73 benchmarks-latencyLinus Torvalds03/16/17 02:01 PM
                  lmbench is horribly brokenWilco03/16/17 04:57 PM
                    lmbench is horribly brokenLinus Torvalds03/16/17 06:49 PM
                      lmbench is horribly brokenLinus Torvalds03/17/17 01:10 PM
                        lmbench is horribly brokenLinus Torvalds03/17/17 01:52 PM
                        lmbench is horribly brokenExophase03/17/17 02:31 PM
                          lmbench is horribly brokenGabriele Svelto03/17/17 03:20 PM
                          lmbench is horribly brokenLinus Torvalds03/17/17 05:56 PM
                            lmbench is horribly brokenExophase03/17/17 06:21 PM
                              lmbench is horribly brokenLinus Torvalds03/17/17 06:43 PM
                                lmbench is horribly brokenIreland03/17/17 07:37 PM
                                  lmbench is horribly brokenbakaneko03/18/17 11:17 AM
                                    lmbench is horribly brokenIreland03/18/17 12:23 PM
                                      lmbench is horribly brokenanon03/18/17 07:35 PM
                                      lmbench is horribly brokenbakaneko03/21/17 08:08 AM
                                        lmbench is horribly brokenIreland03/21/17 03:14 PM
                                lmbench is horribly brokenGabriele Svelto03/18/17 04:01 PM
                                  accessing dram RichardC03/18/17 06:33 PM
                                lmbench is horribly brokenExophase03/18/17 04:26 PM
                                  lmbench is horribly brokenWilco03/18/17 05:40 PM
                                    benchmarking reality?Anon03/19/17 02:29 PM
                                    lmbench is horribly brokenLinus Torvalds03/19/17 04:25 PM
                                      mea culpa (lmbench is horribly broken)Linus Torvalds03/19/17 06:05 PM
                                        mea culpa (lmbench is horribly broken)Bill Broadley03/21/17 01:41 AM
                                          mea culpa (lmbench is horribly broken)Linus Torvalds03/21/17 09:01 AM
                                            mea culpa (lmbench is horribly broken)Linus Torvalds03/21/17 11:14 AM
                                            mea culpa (lmbench is horribly broken)Linus Torvalds03/21/17 05:03 PM
                                              mea culpa (lmbench is horribly broken)Etienne03/22/17 04:37 AM
                                              mea culpa (lmbench is horribly broken)Tim McCaffrey03/22/17 08:54 AM
                                                mea culpa (lmbench is horribly broken)Tim McCaffrey03/22/17 09:34 AM
                                                mea culpa (lmbench is horribly broken)Linus Torvalds03/22/17 10:35 AM
                                                  mea culpa (lmbench is horribly broken)Ireland03/22/17 12:11 PM
                                                    mea culpa (lmbench is horribly broken)Ireland03/22/17 12:26 PM
                                                    mea culpa (lmbench is horribly broken)rwessel03/22/17 03:03 PM
                                                      mea culpa (lmbench is horribly broken)Ireland03/22/17 03:35 PM
                                                  mea culpa (lmbench is horribly broken)Linus Torvalds03/22/17 01:35 PM
                                                    mea culpa (lmbench is horribly broken)Gabriele Svelto03/23/17 08:05 AM
                                                      mea culpa (lmbench is horribly broken)Linus Torvalds03/23/17 10:43 AM
                                                        mea culpa (lmbench is horribly broken)Gabriele Svelto03/23/17 01:56 PM
                                                          mea culpa (lmbench is horribly broken)Ireland03/23/17 02:36 PM
                                                  mea culpa (lmbench is horribly broken)Travis03/22/17 01:38 PM
                                              mea culpa (lmbench is horribly broken)anon03/22/17 07:22 PM
                                                mea culpa (lmbench is horribly broken)Travis03/22/17 08:57 PM
                                                  mea culpa (lmbench is horribly broken)anon03/23/17 12:44 AM
                                                    mea culpa (lmbench is horribly broken)Michael S03/23/17 05:59 PM
                                                      mea culpa (lmbench is horribly broken)Travis03/23/17 09:03 PM
                                              mea culpa (lmbench is horribly broken)anon03/23/17 01:14 AM
                                                mea culpa (lmbench is horribly broken)Linus Torvalds03/23/17 11:22 AM
                                                  Thank you. Associativity misses explain it.anon03/23/17 10:48 PM
                                                    Thank you. Associativity misses explain it.Linus Torvalds03/24/17 01:26 PM
                                                      Thank you. Associativity misses explain it.Ireland03/24/17 02:52 PM
                                  lmbench is horribly brokenLinus Torvalds03/19/17 03:51 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?