Lies, damn lies & SPEC results

Article: Poulson: The Future of Itanium Servers
By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), May 30, 2011 6:04 am
Room: Moderated Discussions
mpx (mpx@nomail.pl) on 5/29/11 wrote:
>
>Why are you mixing the topics of CPU pages, and caching files?

Because the caches aren't just some kernel internal
thing: they are also exposed directly to user space
through mmap.

Now, you can actually consider mmap() to be a separate
issue from caching - and Linux used to do it long ago.
We had a purely-internal "buffer cache" that was used for
disk caching, and then if somebody mmap'ed a file we
would allocate a page for the mmap.

The problem with that is that it's rather complicated,
and very fragile. Especially in the presence of shared
writable mappings (ie user-land actually writing to the
file through a memory mapping), having separate caches
and mappings is really a nasty piece of business.

Fundamental issue #1: cache coherency when you have two
different models and allocations. Not just the CPU caches
for the actual memory, but the coherency at a software
level when one process does a "write()" system call (or
"truncate()" - that really ends up being interesting),
while another one is accessing the thing just by reading
and writing to a mmap'ed area.

Fundamental issue #2: memory use. You actually want to
try to share the backing store memory, because it turns
out that mmap'ed file-backed memory is a big part of your
memory pressure under some loads - big executables. And
you want to share it naturally and easily, not just to
save memory, but because you want to build up and tear
down processes quickly - so you don't want to copy the
executable image around etc.

And don't get me wrong: "nasty piece of business" is not
the same thing as "cannot be done". I'm 100% sure that it
can be done. Can it be done well? I've never seen
it, and in Linux the solution was certainly "don't do
that then", and we unified the caching in the "page cache"
that does everything at a page granularity, rather than
the filesystem block granularity or whatever.

In other words, I've done it both ways, and I can attest
to the fact that personally I would never ever go
back to the bad old days when caching was somehow a
separate issue from the memory mapping support.

So you don't *have* to mix up the MMU page size with the
filesystem cache, but anybody who doesn't is either
incompetent, doesn't know what they are doing, or is
working on some OS that is irrelevant and specialized
and just never needs to worry too much about small
details like mmap.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Poulson Microarchitecture ArticleDavid Kanter2011/05/22 11:47 PM
  Itanium achieved "incredibly high single threaded performance"?Mark Roulo2011/05/23 07:46 AM
    Itanium achieved "incredibly high single threaded performance"?Daniel Bizo2011/05/23 08:08 AM
      Itanium achieved "incredibly high single threaded performance"?David Kanter2011/05/23 08:44 AM
        Itanium achieved "incredibly high single threaded performance"?Michael S2011/05/23 01:20 PM
          Itanium achieved "incredibly high single threaded performance"?Michael S2011/05/23 03:30 PM
        not reallyRichard Cownie2011/05/23 03:37 PM
          you are just spouting anti-IPF propagandaDaniel Bizo2011/05/23 04:10 PM
            you are just spouting anti-IPF propagandaRichard Cownie2011/05/23 05:13 PM
              you are just spouting anti-IPF propagandaLlew2011/05/24 12:04 AM
                you are just spouting anti-IPF propagandaLinus Torvalds2011/05/24 07:58 AM
                  you are just spouting anti-IPF propagandaLlew2011/05/25 01:28 AM
                    you are just spouting anti-IPF propagandaBrett2011/05/25 07:33 AM
                      you are just spouting anti-IPF propagandasomeone2011/05/25 08:22 AM
                        you are just spouting anti-IPF propagandaRakesh Malik2011/05/25 09:25 AM
                          you are just spouting anti-IPF propagandaRichard Cownie2011/05/25 09:56 AM
                            you are just spouting anti-IPF propagandaRakesh Malik2011/05/25 10:06 AM
                              you are just spouting anti-IPF propagandaRichard Cownie2011/05/25 11:13 AM
                                you are just spouting anti-IPF propagandaRakesh Malik2011/05/25 11:22 AM
                                  you are just spouting anti-IPF propagandaMichael S2011/05/26 03:43 AM
                                    POWERx and ItaniumDavid Kanter2011/05/26 04:53 AM
                                    you are just spouting anti-IPF propagandasomeone2011/05/26 08:22 AM
                                      you are just spouting anti-IPF propagandaDavid Kanter2011/05/26 08:40 AM
                                        you are just spouting anti-IPF propagandasomeone2011/05/26 09:11 AM
                                          you are just spouting anti-IPF propagandaMichael S2011/05/26 10:43 AM
                                            Bandwidth in LLCDavid Kanter2011/05/26 03:47 PM
                                              Bandwidth in LLCIntelUser20002011/05/27 12:47 AM
                                              Bandwidth in LLCMichael S2011/05/27 01:45 AM
                                                Bandwidth in LLCDavid Kanter2011/05/27 10:03 AM
                                you are just spouting anti-IPF propagandajrl2011/05/25 06:58 PM
                                  you are just spouting anti-IPF propagandaMichael S2011/05/26 03:28 AM
                                    obsess much?someone2011/05/26 08:14 AM
                                      obsess much?Michael S2011/05/26 09:56 AM
                                        obsess much?someone2011/05/26 12:00 PM
                                    you are just spouting anti-IPF propagandajrl2011/05/27 07:46 PM
                                      you are just spouting anti-IPF propagandaanonymous2011/05/27 11:03 PM
                        you are just spouting anti-IPF propagandaBrett2011/05/25 11:56 AM
                          you are just spouting anti-IPF propagandaDavid Kanter2011/05/25 02:00 PM
                            you are just spouting anti-IPF propagandaRichard Cownie2011/05/25 03:23 PM
                              Large cachesDavid Kanter2011/05/25 06:22 PM
                                Large cachesRichard Cownie2011/05/25 07:18 PM
                                  Large cachesanon2011/05/26 05:03 PM
                                    Large cachesRichard Cownie2011/05/26 05:50 PM
                                      Large cachesanon2011/05/26 06:36 PM
                                        Large cachesRichard Cownie2011/05/26 07:46 PM
                                          Large cachesanon2011/05/26 11:43 PM
                                            Large cachesRichard Cownie2011/05/27 02:24 AM
                                              Large cachesDavid Kanter2011/05/27 10:20 AM
                                                Large cachesRichard Cownie2011/05/27 10:51 AM
                                                  Large cachesAaron Spink2011/05/27 11:18 AM
                                                    Large cachesRichard Cownie2011/05/27 11:43 AM
                                                      Large cachessomeone2011/05/27 01:00 PM
                                                        Large cachesRichard Cownie2011/05/27 04:59 PM
                                                          WorkstationsDavid Kanter2011/05/27 08:25 PM
                                                            WorkstationsRichard Cownie2011/05/28 05:21 AM
                                                              WorkstationsMichael S2011/05/29 03:25 AM
                                                          Large cachesAaron Spink2011/05/27 10:11 PM
                                                    Large cachesCarlie Coats2011/05/27 01:33 PM
                                                      Large cachesAaron Spink2011/05/27 02:28 PM
                                              Large cachesanon2011/05/27 05:39 PM
                                          Large cachesAaron Spink2011/05/27 11:11 AM
                                            Large cachesLinus Torvalds2011/05/27 01:38 PM
                                              Large cachesAaron Spink2011/05/27 02:33 PM
                                                Large cachesLinus Torvalds2011/05/27 04:26 PM
                                                  Miss distanceDavid Kanter2011/05/27 08:01 PM
                                                    Miss distanceLlew2011/05/27 11:27 PM
                                                      Miss distanceanon2011/05/28 12:00 AM
                                                      Miss distancerwessel2011/05/28 01:01 AM
                                                        Miss distanceLlew2011/05/28 02:14 AM
                                                    Miss distanceLinus Torvalds2011/05/28 07:55 AM
                                                      Miss distancea reader2011/05/28 03:50 PM
                                                        Miss distanceLinus Torvalds2011/05/28 04:08 PM
                                                          Miss distancea reader2011/05/29 07:32 AM
                                                    Miss distanceajensen2011/05/29 12:14 AM
                                                      Miss distanceLlew2011/05/29 02:01 AM
                                                      Miss distancebakaneko2011/05/29 04:20 AM
                                                  Run-ahead gives MLP for IOES. Rao2011/05/28 05:45 PM
                                                    Run-ahead gives MLP for IOELinus Torvalds2011/05/28 08:53 PM
                                                      Run-ahead gives MLP for IOES. Rao2011/05/28 09:31 PM
                                                        Run-ahead gives MLP for IOELlew2011/05/29 12:03 AM
                                                          Run-ahead gives MLP for IOEAaron Spink2011/05/29 03:03 AM
                                                            Run-ahead gives MLP for IOEa reader2011/05/29 07:27 AM
                                        Large cachesDavid Kanter2011/05/26 09:58 PM
                                          Large cachesanon2011/05/26 11:41 PM
                                            Large cachesRichard Cownie2011/05/27 05:46 AM
                                              Large cachesanon2011/05/27 10:07 PM
                                Large cachesBrett2011/05/26 09:28 AM
                                  Large cachesDan Downs2011/05/26 01:01 PM
                                    Large cachesDavid Kanter2011/05/26 03:52 PM
                                      Large cachesMichael S2011/05/27 02:01 AM
                                    Large cachesBrett2011/05/26 08:38 PM
                                      Large cachesDan Downs2011/05/27 10:59 AM
                                        Large cachesBrett2011/05/28 10:45 AM
                            you are just spouting anti-IPF propagandampx2011/05/26 12:57 PM
                        you are just spouting anti-IPF propagandaMichael S2011/05/26 04:01 AM
                          you are just spouting anti-IPF propagandaanon2011/05/26 05:21 PM
                      you are just spouting anti-IPF propagandaanon2011/05/25 09:13 AM
                  you are just spouting anti-IPF propagandaDavid Kanter2011/05/26 12:45 AM
                    you are just spouting anti-IPF propagandasomeone2011/05/26 08:55 AM
                      Poulson power and perfDavid Kanter2011/05/26 09:10 AM
                        Poulson power and perfsomeone2011/05/26 09:40 AM
                        Poulson power and perfanon2011/05/26 09:31 PM
                          Poulson power and perfDavid Kanter2011/05/26 10:12 PM
                            Poulson power and perfanon2011/05/28 08:24 AM
                              Poulson power and perfDavid Kanter2011/05/28 10:28 AM
              you are just spouting anti-IPF propagandaDaniel Bizo2011/05/24 12:30 AM
                you are just spouting anti-IPF propagandaRichard Cownie2011/05/24 02:28 AM
                you are just spouting anti-IPF propagandampx2011/05/25 12:03 PM
                  you are just spouting anti-IPF propagandaMichael S2011/05/26 03:06 AM
                    you are just spouting anti-IPF propagandampx2011/05/26 12:27 PM
            highest performing xx nm CPUsMichael S2011/05/24 08:55 AM
      Itanium achieved "incredibly high single threaded performance"?Max2011/05/23 10:07 AM
        Itanium achieved "incredibly high single threaded performance"?Daniel Bizo2011/05/23 01:59 PM
    Itanium achieved "incredibly high single threaded performance"?David Kanter2011/05/23 08:14 AM
      Itanium achieved "incredibly high single threaded performance"?Michael S2011/05/23 01:17 PM
        Itanium achieved "incredibly high single threaded performance"?David Kanter2011/05/23 05:56 PM
          Itanium achieved "incredibly high single threaded performance"?Thu Nguyen2011/05/23 09:09 PM
            Itanium achieved "incredibly high single threaded performance"?David Kanter2011/05/23 10:11 PM
              Itanium achieved "incredibly high single threaded performance"?Michael S2011/05/24 03:08 AM
                Itanium achieved "incredibly high single threaded performance"?David Kanter2011/05/24 08:36 AM
                  Itanium achieved "incredibly high single threaded performance"?Michael S2011/05/24 09:17 AM
                  Itanium achieved "incredibly high single threaded performance"?Thu Nguyen2011/05/24 01:45 PM
                    POWER7 vs. SNB SPECint_baseDavid Kanter2011/05/24 09:15 PM
                      SNB SPECint linkDavid Kanter2011/05/24 09:17 PM
                      Lies, damn lies & SPEC resultsPassing by2011/05/26 07:06 AM
                        Lies, damn lies & SPEC resultsDavid Kanter2011/05/26 08:17 AM
                          Lies, damn lies & SPEC resultsMichael S2011/05/26 10:57 AM
                          Lies, damn lies & SPEC resultsanon2011/05/26 05:44 PM
                        Lies, damn lies & SPEC resultsMichael S2011/05/26 11:04 AM
                          Lies, damn lies & SPEC resultsLinus Torvalds2011/05/26 11:53 AM
                            Lies, damn lies & SPEC resultsJouni Osmala2011/05/27 12:42 AM
                              mem usage distributionMichael S2011/05/27 01:38 AM
                                mem usage distributionJouni Osmala2011/05/27 04:25 AM
                              Lies, damn lies & SPEC resultsLinus Torvalds2011/05/27 06:44 AM
                                Lies, damn lies & SPEC resultsJouni Osmala2011/05/28 05:19 AM
                                  Lies, damn lies & SPEC resultsLinus Torvalds2011/05/28 08:29 AM
                                    Using larger translationsPaul A. Clayton2011/05/28 12:17 PM
                                      Using larger translationsLinus Torvalds2011/05/28 12:58 PM
                                        Using larger translationsanon2011/05/28 04:37 PM
                                        Large pages still seem attractivePaul A. Clayton2011/05/28 08:47 PM
                                          Large pages still seem attractiveLinus Torvalds2011/05/28 10:02 PM
                                    Lies, damn lies & SPEC resultsDoug Siebert2011/05/28 12:35 PM
                                      Lies, damn lies & SPEC resultsLinus Torvalds2011/05/28 01:14 PM
                                        Lies, damn lies & SPEC resultsLinus Torvalds2011/05/28 01:25 PM
                                        Lies, damn lies & SPEC resultsmpx2011/05/29 08:12 AM
                                          Lies, damn lies & SPEC resultsLinus Torvalds2011/05/30 06:04 AM
                                            Lies, damn lies & SPEC resultsgallier22011/05/30 07:23 AM
                                            Lies, damn lies & SPEC resultsmpx2011/05/30 01:49 PM
                                        Lies, damn lies & SPEC resultsDoug Siebert2011/05/29 11:30 AM
                                      Lies, damn lies & SPEC resultsanon2011/05/28 03:54 PM
                                        Lies, damn lies & SPEC resultsnone2011/05/28 04:58 PM
                                          How's that for compatibility?David Kanter2011/05/28 05:14 PM
                                            How's that for compatibility?none2011/05/28 05:38 PM
                                              How's that for compatibility?anon2011/05/28 08:00 PM
                                                How's that for compatibility?Jouni Osmala2011/05/28 11:32 PM
                                                  Larger caches with smaller pagesPaul A. Clayton2011/05/29 08:40 PM
                                                How's that for compatibility?Wilco2011/05/30 10:18 AM
                                                  How's that for compatibility?anon2011/05/30 10:20 PM
                                                    How's that for compatibility?Wilco2011/06/01 03:47 AM
                                                      How's that for compatibility?anon2011/06/01 05:12 AM
                                                        How's that for compatibility?Michael S2011/06/02 01:41 AM
                                                          How's that for compatibility?Wilco2011/06/02 03:01 AM
                                                            How's that for compatibility?none2011/06/02 09:03 AM
                                      Lies, damn lies & SPEC resultsanon2011/05/28 06:02 PM
                                        Lies, damn lies & SPEC resultsDavid Kanter2011/05/28 07:21 PM
                                        Lies, damn lies & SPEC resultsS. Rao2011/05/28 07:34 PM
                                          Lies, damn lies & SPEC resultsanon2011/05/28 11:47 PM
                                            Lies, damn lies & SPEC resultsS. Rao2011/05/29 01:09 AM
                                              Lies, damn lies & SPEC resultsMichael S2011/05/29 02:59 AM
                                                Lies, damn lies & SPEC resultsanon2011/05/29 03:45 AM
                                                  Lies, damn lies & SPEC resultsanon2011/05/29 04:18 AM
                                            Lies, damn lies & SPEC resultsJouni Osmala2011/05/29 01:49 AM
                                              Lies, damn lies & SPEC resultsanon2011/05/29 03:32 AM
                                                Lies, damn lies & SPEC resultsJouni Osmala2011/05/29 04:19 AM
                                                  Lies, damn lies & SPEC resultsMichael S2011/05/29 09:05 AM
                                                    Lies, damn lies & SPEC resultsJouni Osmala2011/05/29 09:40 AM
                                                      Lies, damn lies & SPEC resultsMichael S2011/05/29 10:25 AM
                                                  Lies, damn lies & SPEC resultsanon2011/05/29 11:50 AM
                                              Lies, damn lies & SPEC resultsmpx2011/05/30 03:22 AM
                                      Lies, damn lies & SPEC resultsGabriele Svelto2011/05/29 02:09 AM
                          Lies, damn lies & SPEC resultsanon2011/05/26 05:58 PM
                  Itanium achieved "incredibly high single threaded performance"?anon2011/05/24 10:25 PM
    Itanium achieved "incredibly high single threaded performance"?someone2011/05/24 06:40 AM
      Itanium achieved "incredibly high single threaded performance"?none2011/05/24 08:08 AM
        Itanium achieved "incredibly high single threaded performance"?someone2011/05/24 09:35 AM
      Itanium achieved "incredibly high single threaded performance"?Megol2011/05/24 08:10 AM
      Itanium achieved "incredibly high single threaded performance"?Seni2011/05/24 08:43 AM
        Itanium achieved "incredibly high single threaded performance"?someone2011/05/24 09:38 AM
      Itanium achieved "incredibly high single threaded performance"?Mark Roulo2011/05/24 02:41 PM
        P4 180 nm with L3anonymous2011/05/25 08:16 AM
      Itanium achieved "incredibly high single threaded performance"?TA152H2011/05/25 09:58 AM
  Thanks!Wouter Tinus2011/05/23 11:21 AM
  Poulson Microarchitecture Articleajensen2011/05/23 12:16 PM
    Poulson Microarchitecture ArticleDavid Kanter2011/05/23 05:48 PM
      Poulson control queueajensen2011/05/23 09:48 PM
        Poulson control queueDavid Kanter2011/05/26 12:22 AM
  Some nitpicks and comments/questionsPaul A. Clayton2011/05/23 12:22 PM
    Wide-Word [was: Some nitpicks and comments/questions]anon2011/05/23 08:33 PM
    Some nitpicks and comments/questionsDavid Kanter2011/05/24 09:36 AM
      Little technically keeps Xeon from true high-endPaul A. Clayton2011/05/24 03:56 PM
        Little technically keeps Xeon from true high-endDavid Hess2011/05/24 07:16 PM
          ChipKill implementation requirementsPaul A. Clayton2011/05/25 08:38 AM
            ChipKill implementation requirementsDavid Hess2011/05/25 05:10 PM
  spec int base referenceajensen2011/05/24 01:08 PM
    spec int base referenceMichael S2011/05/24 02:08 PM
    spec int base referenceMichael S2011/05/24 03:02 PM
      spec int base referenceajensen2011/05/24 10:38 PM
        spec int base referenceMichael S2011/05/25 01:44 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?