Lies, damn lies & SPEC results

Article: Poulson: The Future of Itanium Servers
By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), May 30, 2011 6:04 am
Room: Moderated Discussions
mpx (mpx@nomail.pl) on 5/29/11 wrote:
>
>Why are you mixing the topics of CPU pages, and caching files?

Because the caches aren't just some kernel internal
thing: they are also exposed directly to user space
through mmap.

Now, you can actually consider mmap() to be a separate
issue from caching - and Linux used to do it long ago.
We had a purely-internal "buffer cache" that was used for
disk caching, and then if somebody mmap'ed a file we
would allocate a page for the mmap.

The problem with that is that it's rather complicated,
and very fragile. Especially in the presence of shared
writable mappings (ie user-land actually writing to the
file through a memory mapping), having separate caches
and mappings is really a nasty piece of business.

Fundamental issue #1: cache coherency when you have two
different models and allocations. Not just the CPU caches
for the actual memory, but the coherency at a software
level when one process does a "write()" system call (or
"truncate()" - that really ends up being interesting),
while another one is accessing the thing just by reading
and writing to a mmap'ed area.

Fundamental issue #2: memory use. You actually want to
try to share the backing store memory, because it turns
out that mmap'ed file-backed memory is a big part of your
memory pressure under some loads - big executables. And
you want to share it naturally and easily, not just to
save memory, but because you want to build up and tear
down processes quickly - so you don't want to copy the
executable image around etc.

And don't get me wrong: "nasty piece of business" is not
the same thing as "cannot be done". I'm 100% sure that it
can be done. Can it be done well? I've never seen
it, and in Linux the solution was certainly "don't do
that then", and we unified the caching in the "page cache"
that does everything at a page granularity, rather than
the filesystem block granularity or whatever.

In other words, I've done it both ways, and I can attest
to the fact that personally I would never ever go
back to the bad old days when caching was somehow a
separate issue from the memory mapping support.

So you don't *have* to mix up the MMU page size with the
filesystem cache, but anybody who doesn't is either
incompetent, doesn't know what they are doing, or is
working on some OS that is irrelevant and specialized
and just never needs to worry too much about small
details like mmap.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Poulson Microarchitecture ArticleDavid Kanter05/22/11 11:47 PM
  Itanium achieved "incredibly high single threaded performance"?Mark Roulo05/23/11 07:46 AM
    Itanium achieved "incredibly high single threaded performance"?Daniel Bizo05/23/11 08:08 AM
      Itanium achieved "incredibly high single threaded performance"?David Kanter05/23/11 08:44 AM
        Itanium achieved "incredibly high single threaded performance"?Michael S05/23/11 01:20 PM
          Itanium achieved "incredibly high single threaded performance"?Michael S05/23/11 03:30 PM
        not reallyRichard Cownie05/23/11 03:37 PM
          you are just spouting anti-IPF propagandaDaniel Bizo05/23/11 04:10 PM
            you are just spouting anti-IPF propagandaRichard Cownie05/23/11 05:13 PM
              you are just spouting anti-IPF propagandaLlew05/24/11 12:04 AM
                you are just spouting anti-IPF propagandaLinus Torvalds05/24/11 07:58 AM
                  you are just spouting anti-IPF propagandaLlew05/25/11 01:28 AM
                    you are just spouting anti-IPF propagandaBrett05/25/11 07:33 AM
                      you are just spouting anti-IPF propagandasomeone05/25/11 08:22 AM
                        you are just spouting anti-IPF propagandaRakesh Malik05/25/11 09:25 AM
                          you are just spouting anti-IPF propagandaRichard Cownie05/25/11 09:56 AM
                            you are just spouting anti-IPF propagandaRakesh Malik05/25/11 10:06 AM
                              you are just spouting anti-IPF propagandaRichard Cownie05/25/11 11:13 AM
                                you are just spouting anti-IPF propagandaRakesh Malik05/25/11 11:22 AM
                                  you are just spouting anti-IPF propagandaMichael S05/26/11 03:43 AM
                                    POWERx and ItaniumDavid Kanter05/26/11 04:53 AM
                                    you are just spouting anti-IPF propagandasomeone05/26/11 08:22 AM
                                      you are just spouting anti-IPF propagandaDavid Kanter05/26/11 08:40 AM
                                        you are just spouting anti-IPF propagandasomeone05/26/11 09:11 AM
                                          you are just spouting anti-IPF propagandaMichael S05/26/11 10:43 AM
                                            Bandwidth in LLCDavid Kanter05/26/11 03:47 PM
                                              Bandwidth in LLCIntelUser200005/27/11 12:47 AM
                                              Bandwidth in LLCMichael S05/27/11 01:45 AM
                                                Bandwidth in LLCDavid Kanter05/27/11 10:03 AM
                                you are just spouting anti-IPF propagandajrl05/25/11 06:58 PM
                                  you are just spouting anti-IPF propagandaMichael S05/26/11 03:28 AM
                                    obsess much?someone05/26/11 08:14 AM
                                      obsess much?Michael S05/26/11 09:56 AM
                                        obsess much?someone05/26/11 12:00 PM
                                    you are just spouting anti-IPF propagandajrl05/27/11 07:46 PM
                                      you are just spouting anti-IPF propagandaanonymous05/27/11 11:03 PM
                        you are just spouting anti-IPF propagandaBrett05/25/11 11:56 AM
                          you are just spouting anti-IPF propagandaDavid Kanter05/25/11 02:00 PM
                            you are just spouting anti-IPF propagandaRichard Cownie05/25/11 03:23 PM
                              Large cachesDavid Kanter05/25/11 06:22 PM
                                Large cachesRichard Cownie05/25/11 07:18 PM
                                  Large cachesanon05/26/11 05:03 PM
                                    Large cachesRichard Cownie05/26/11 05:50 PM
                                      Large cachesanon05/26/11 06:36 PM
                                        Large cachesRichard Cownie05/26/11 07:46 PM
                                          Large cachesanon05/26/11 11:43 PM
                                            Large cachesRichard Cownie05/27/11 02:24 AM
                                              Large cachesDavid Kanter05/27/11 10:20 AM
                                                Large cachesRichard Cownie05/27/11 10:51 AM
                                                  Large cachesAaron Spink05/27/11 11:18 AM
                                                    Large cachesRichard Cownie05/27/11 11:43 AM
                                                      Large cachessomeone05/27/11 01:00 PM
                                                        Large cachesRichard Cownie05/27/11 04:59 PM
                                                          WorkstationsDavid Kanter05/27/11 08:25 PM
                                                            WorkstationsRichard Cownie05/28/11 05:21 AM
                                                              WorkstationsMichael S05/29/11 03:25 AM
                                                          Large cachesAaron Spink05/27/11 10:11 PM
                                                    Large cachesCarlie Coats05/27/11 01:33 PM
                                                      Large cachesAaron Spink05/27/11 02:28 PM
                                              Large cachesanon05/27/11 05:39 PM
                                          Large cachesAaron Spink05/27/11 11:11 AM
                                            Large cachesLinus Torvalds05/27/11 01:38 PM
                                              Large cachesAaron Spink05/27/11 02:33 PM
                                                Large cachesLinus Torvalds05/27/11 04:26 PM
                                                  Miss distanceDavid Kanter05/27/11 08:01 PM
                                                    Miss distanceLlew05/27/11 11:27 PM
                                                      Miss distanceanon05/28/11 12:00 AM
                                                      Miss distancerwessel05/28/11 01:01 AM
                                                        Miss distanceLlew05/28/11 02:14 AM
                                                    Miss distanceLinus Torvalds05/28/11 07:55 AM
                                                      Miss distancea reader05/28/11 03:50 PM
                                                        Miss distanceLinus Torvalds05/28/11 04:08 PM
                                                          Miss distancea reader05/29/11 07:32 AM
                                                    Miss distanceajensen05/29/11 12:14 AM
                                                      Miss distanceLlew05/29/11 02:01 AM
                                                      Miss distancebakaneko05/29/11 04:20 AM
                                                  Run-ahead gives MLP for IOES. Rao05/28/11 05:45 PM
                                                    Run-ahead gives MLP for IOELinus Torvalds05/28/11 08:53 PM
                                                      Run-ahead gives MLP for IOES. Rao05/28/11 09:31 PM
                                                        Run-ahead gives MLP for IOELlew05/29/11 12:03 AM
                                                          Run-ahead gives MLP for IOEAaron Spink05/29/11 03:03 AM
                                                            Run-ahead gives MLP for IOEa reader05/29/11 07:27 AM
                                        Large cachesDavid Kanter05/26/11 09:58 PM
                                          Large cachesanon05/26/11 11:41 PM
                                            Large cachesRichard Cownie05/27/11 05:46 AM
                                              Large cachesanon05/27/11 10:07 PM
                                Large cachesBrett05/26/11 09:28 AM
                                  Large cachesDan Downs05/26/11 01:01 PM
                                    Large cachesDavid Kanter05/26/11 03:52 PM
                                      Large cachesMichael S05/27/11 02:01 AM
                                    Large cachesBrett05/26/11 08:38 PM
                                      Large cachesDan Downs05/27/11 10:59 AM
                                        Large cachesBrett05/28/11 10:45 AM
                            you are just spouting anti-IPF propagandampx05/26/11 12:57 PM
                        you are just spouting anti-IPF propagandaMichael S05/26/11 04:01 AM
                          you are just spouting anti-IPF propagandaanon05/26/11 05:21 PM
                      you are just spouting anti-IPF propagandaanon05/25/11 09:13 AM
                  you are just spouting anti-IPF propagandaDavid Kanter05/26/11 12:45 AM
                    you are just spouting anti-IPF propagandasomeone05/26/11 08:55 AM
                      Poulson power and perfDavid Kanter05/26/11 09:10 AM
                        Poulson power and perfsomeone05/26/11 09:40 AM
                        Poulson power and perfanon05/26/11 09:31 PM
                          Poulson power and perfDavid Kanter05/26/11 10:12 PM
                            Poulson power and perfanon05/28/11 08:24 AM
                              Poulson power and perfDavid Kanter05/28/11 10:28 AM
              you are just spouting anti-IPF propagandaDaniel Bizo05/24/11 12:30 AM
                you are just spouting anti-IPF propagandaRichard Cownie05/24/11 02:28 AM
                you are just spouting anti-IPF propagandampx05/25/11 12:03 PM
                  you are just spouting anti-IPF propagandaMichael S05/26/11 03:06 AM
                    you are just spouting anti-IPF propagandampx05/26/11 12:27 PM
            highest performing xx nm CPUsMichael S05/24/11 08:55 AM
      Itanium achieved "incredibly high single threaded performance"?Max05/23/11 10:07 AM
        Itanium achieved "incredibly high single threaded performance"?Daniel Bizo05/23/11 01:59 PM
    Itanium achieved "incredibly high single threaded performance"?David Kanter05/23/11 08:14 AM
      Itanium achieved "incredibly high single threaded performance"?Michael S05/23/11 01:17 PM
        Itanium achieved "incredibly high single threaded performance"?David Kanter05/23/11 05:56 PM
          Itanium achieved "incredibly high single threaded performance"?Thu Nguyen05/23/11 09:09 PM
            Itanium achieved "incredibly high single threaded performance"?David Kanter05/23/11 10:11 PM
              Itanium achieved "incredibly high single threaded performance"?Michael S05/24/11 03:08 AM
                Itanium achieved "incredibly high single threaded performance"?David Kanter05/24/11 08:36 AM
                  Itanium achieved "incredibly high single threaded performance"?Michael S05/24/11 09:17 AM
                  Itanium achieved "incredibly high single threaded performance"?Thu Nguyen05/24/11 01:45 PM
                    POWER7 vs. SNB SPECint_baseDavid Kanter05/24/11 09:15 PM
                      SNB SPECint linkDavid Kanter05/24/11 09:17 PM
                      Lies, damn lies & SPEC resultsPassing by05/26/11 07:06 AM
                        Lies, damn lies & SPEC resultsDavid Kanter05/26/11 08:17 AM
                          Lies, damn lies & SPEC resultsMichael S05/26/11 10:57 AM
                          Lies, damn lies & SPEC resultsanon05/26/11 05:44 PM
                        Lies, damn lies & SPEC resultsMichael S05/26/11 11:04 AM
                          Lies, damn lies & SPEC resultsLinus Torvalds05/26/11 11:53 AM
                            Lies, damn lies & SPEC resultsJouni Osmala05/27/11 12:42 AM
                              mem usage distributionMichael S05/27/11 01:38 AM
                                mem usage distributionJouni Osmala05/27/11 04:25 AM
                              Lies, damn lies & SPEC resultsLinus Torvalds05/27/11 06:44 AM
                                Lies, damn lies & SPEC resultsJouni Osmala05/28/11 05:19 AM
                                  Lies, damn lies & SPEC resultsLinus Torvalds05/28/11 08:29 AM
                                    Using larger translationsPaul A. Clayton05/28/11 12:17 PM
                                      Using larger translationsLinus Torvalds05/28/11 12:58 PM
                                        Using larger translationsanon05/28/11 04:37 PM
                                        Large pages still seem attractivePaul A. Clayton05/28/11 08:47 PM
                                          Large pages still seem attractiveLinus Torvalds05/28/11 10:02 PM
                                    Lies, damn lies & SPEC resultsDoug Siebert05/28/11 12:35 PM
                                      Lies, damn lies & SPEC resultsLinus Torvalds05/28/11 01:14 PM
                                        Lies, damn lies & SPEC resultsLinus Torvalds05/28/11 01:25 PM
                                        Lies, damn lies & SPEC resultsmpx05/29/11 08:12 AM
                                          Lies, damn lies & SPEC resultsLinus Torvalds05/30/11 06:04 AM
                                            Lies, damn lies & SPEC resultsgallier205/30/11 07:23 AM
                                            Lies, damn lies & SPEC resultsmpx05/30/11 01:49 PM
                                        Lies, damn lies & SPEC resultsDoug Siebert05/29/11 11:30 AM
                                      Lies, damn lies & SPEC resultsanon05/28/11 03:54 PM
                                        Lies, damn lies & SPEC resultsnone05/28/11 04:58 PM
                                          How's that for compatibility?David Kanter05/28/11 05:14 PM
                                            How's that for compatibility?none05/28/11 05:38 PM
                                              How's that for compatibility?anon05/28/11 08:00 PM
                                                How's that for compatibility?Jouni Osmala05/28/11 11:32 PM
                                                  Larger caches with smaller pagesPaul A. Clayton05/29/11 08:40 PM
                                                How's that for compatibility?Wilco05/30/11 10:18 AM
                                                  How's that for compatibility?anon05/30/11 10:20 PM
                                                    How's that for compatibility?Wilco06/01/11 03:47 AM
                                                      How's that for compatibility?anon06/01/11 05:12 AM
                                                        How's that for compatibility?Michael S06/02/11 01:41 AM
                                                          How's that for compatibility?Wilco06/02/11 03:01 AM
                                                            How's that for compatibility?none06/02/11 09:03 AM
                                      Lies, damn lies & SPEC resultsanon05/28/11 06:02 PM
                                        Lies, damn lies & SPEC resultsDavid Kanter05/28/11 07:21 PM
                                        Lies, damn lies & SPEC resultsS. Rao05/28/11 07:34 PM
                                          Lies, damn lies & SPEC resultsanon05/28/11 11:47 PM
                                            Lies, damn lies & SPEC resultsS. Rao05/29/11 01:09 AM
                                              Lies, damn lies & SPEC resultsMichael S05/29/11 02:59 AM
                                                Lies, damn lies & SPEC resultsanon05/29/11 03:45 AM
                                                  Lies, damn lies & SPEC resultsanon05/29/11 04:18 AM
                                            Lies, damn lies & SPEC resultsJouni Osmala05/29/11 01:49 AM
                                              Lies, damn lies & SPEC resultsanon05/29/11 03:32 AM
                                                Lies, damn lies & SPEC resultsJouni Osmala05/29/11 04:19 AM
                                                  Lies, damn lies & SPEC resultsMichael S05/29/11 09:05 AM
                                                    Lies, damn lies & SPEC resultsJouni Osmala05/29/11 09:40 AM
                                                      Lies, damn lies & SPEC resultsMichael S05/29/11 10:25 AM
                                                  Lies, damn lies & SPEC resultsanon05/29/11 11:50 AM
                                              Lies, damn lies & SPEC resultsmpx05/30/11 03:22 AM
                                      Lies, damn lies & SPEC resultsGabriele Svelto05/29/11 02:09 AM
                          Lies, damn lies & SPEC resultsanon05/26/11 05:58 PM
                  Itanium achieved "incredibly high single threaded performance"?anon05/24/11 10:25 PM
    Itanium achieved "incredibly high single threaded performance"?someone05/24/11 06:40 AM
      Itanium achieved "incredibly high single threaded performance"?none05/24/11 08:08 AM
        Itanium achieved "incredibly high single threaded performance"?someone05/24/11 09:35 AM
      Itanium achieved "incredibly high single threaded performance"?Megol05/24/11 08:10 AM
      Itanium achieved "incredibly high single threaded performance"?Seni05/24/11 08:43 AM
        Itanium achieved "incredibly high single threaded performance"?someone05/24/11 09:38 AM
      Itanium achieved "incredibly high single threaded performance"?Mark Roulo05/24/11 02:41 PM
        P4 180 nm with L3anonymous05/25/11 08:16 AM
      Itanium achieved "incredibly high single threaded performance"?TA152H05/25/11 09:58 AM
  Thanks!Wouter Tinus05/23/11 11:21 AM
  Poulson Microarchitecture Articleajensen05/23/11 12:16 PM
    Poulson Microarchitecture ArticleDavid Kanter05/23/11 05:48 PM
      Poulson control queueajensen05/23/11 09:48 PM
        Poulson control queueDavid Kanter05/26/11 12:22 AM
  Some nitpicks and comments/questionsPaul A. Clayton05/23/11 12:22 PM
    Wide-Word [was: Some nitpicks and comments/questions]anon05/23/11 08:33 PM
    Some nitpicks and comments/questionsDavid Kanter05/24/11 09:36 AM
      Little technically keeps Xeon from true high-endPaul A. Clayton05/24/11 03:56 PM
        Little technically keeps Xeon from true high-endDavid Hess05/24/11 07:16 PM
          ChipKill implementation requirementsPaul A. Clayton05/25/11 08:38 AM
            ChipKill implementation requirementsDavid Hess05/25/11 05:10 PM
  spec int base referenceajensen05/24/11 01:08 PM
    spec int base referenceMichael S05/24/11 02:08 PM
    spec int base referenceMichael S05/24/11 03:02 PM
      spec int base referenceajensen05/24/11 10:38 PM
        spec int base referenceMichael S05/25/11 01:44 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell blue?