Sharing between servers and phones

By: rwessel (, January 29, 2017 8:56 pm
Room: Moderated Discussions
Linus Torvalds ( on January 28, 2017 11:49 am wrote:
> David Kanter ( on January 27, 2017 6:46 am wrote:
> >
> > The TLB is a good example. It is larger in servers do deal with
> > larger data footprints. But TLBs are power-hungry and large.
> I'm not actually sure that the TLB is all that good an example.
> There are various trade-offs wrt TLB's, and "larger data footprints" is only one issue.
> Another (big) issue for TLB's is interface and usage, and latency of the
> inevitable misses (and lots of misses are compulsory, not capacity).
> To be specific, for Intel the traditional TLB lookup model was (and that traditional
> model still affects things today, even if it's slightly modified):
> (a) no address space ID's
> (b) very frequent TLB flushes
> where (a) is simply from historical interface reasons and (b) is from how the traditional win32 GDI subsystem
> worked (and other workarounds for things like PAE) which just caused the TLB to be flushed all the time.
> Now, the naive assumption is that "if the TLB gets flushed very often, then a
> small TLB makes sense". But that's not actually really all that happened.
> What happened is that because of the constant TLB flushes, the TLB re-population simply got a lot more
> critical for Intel (and AMD) CPU's than it tends to be for the "server only" CPU's that you mention.
> If you are a POWER architect, the easy way to make the TLB more effective is to
> just make it much bigger. You are running only big jobs that almost never flush the
> TLB, so making the TLB bigger is a simply no-brainer. Same is true on zArch.
> But I really want to stress that "no brainer" part. It's not a "clever" approach. It's also not something
> that should be seen as a good thing and be seen as "big box serious hardware for real men and women".
> It's the stupid approach that just happens to work pretty well under the kinds of boring and big
> - but otherwise pretty well-behaved - applications that those CPU's were mostly running.
> The thing is, the POWER page tables are traditionally horrible nasty shit, and TLB misses take a long time and
> need software support to fill in the hashed page tables. The thing is just bad. It's a really bad design,
> and the big TLB's are pure brawn with no brains. I'm not as familiar with s390, so I'm not going to say horrible
> things about that, but I do want to stress that "big TLB" is not necessarily a good thing in itself.
> Because the other side of the coin really is how well you can fill that TLB.
> And if you are better at filling the TLB, you may simply need a smaller TLB in practice.
> Put another way: I'd rather take a smaller, smarter, low-latency cache that is coupled
> with a smart memory subsystem that can have multiple outstanding misses and does a good
> job of prefetching and not stalling the pipeline in inconvenient situations.
> And I think you'd say "Duh! Of course" if you thought about it from the standpoint of the regular L1D$.
> I'm saying that the exact same thing is true of the TLB. Size is absolutely not
> everything. "Big" does not automatically equal "good". If everything else is equal,
> big is obviously better, but everything else is very much not equal.
> Server loads are in many ways simpler than desktop and mobile loads. A lot of traditional server
> loads can be handled by just putting "more" of everything. More cores. Bigger caches and TLB's.
> More memory. More, more, more. But seldom "clever". Brute force over nimble and smart. I don't
> think people call some of those things dinosaurs just because they are old.

That's at least partially untrue for zArch. Context switches are very common in MVS*, as many things happen via calls to other address spaces (most often mediated by the zArch equivalent of a call gate**). zArch's TLBs (and specifically the two level TLBs) are there to avoid needing TLB reloads precisely because address space context switches have been increasing significantly in frequency for the last three decades.

Much of GDI in Widnows was moved back into the kernel, back in the NT4 days (although some has moved back out), because the resulting number of micro-kernel-ish address space context switches was too painful. But in general, efficient cross-address space calls has a major issue for MVS for decades.

As to how the page tables are organized, it's very little like POWER, and quite similar to x86 in gross terms (every detail is different, of course). It's a fairly hierarchical table of 2-5 levels (allowing 2GB, 4TB, 8PB or 16EB address spaces - the upper four levels each map 11 bits, the lowest 8 bits, mainly for historical reasons***). There is some variability in what each entry maps (a higher level table can specify that the lower level table its referencing only covers a subset of its nominal range - for example, a second level table can specify that the subordinate third level table only maps the second and third pages, instead of the four it would nominally contain, thus providing translations only for the middle two quarters of the region). There are some other details, not least a specific tag for common (shared) table entries ("segments"). The TLB is defined so that the origin of the page table is essentially the ASID, so translations from multiple address spaces can coexist in the TLB, and TLB entries, at least for the lower levels, that come from the same physical page are largely assumed to be common. That obviously impacts code that needs to invalidate TLB entries, since the TLB can retain entries for tables not currently attached. A variety of flush instructions are provided. There is, of course, a fair bit of other cruft in there, as well as ties to the address space authorization and call gate mechanisms.

While fast TLB reloads are certainly a good thing, they are still an expense, avoiding unnecessary discards of TLB entries, especially on short switches to different address spaces, compliments that. Just like not flushing an L1 for a short call to a different address space is good, no matter how fast the reloads are.

Virtualization also benefits from the large TLBs, again, avoiding discarding a lot of TLB data on VM switches.

*Yeah, yeah, zOS. Just be glad I've stopping calling it Ozzy.

**Unlike most attempts to provide a call-gate-like function, zArch call-gates are actually quite heavily used in MVS system and application code (in application code usually hidden in a library). OTOH, the S/370 "program call" mechanism (and its evolutions under XA, ESA and zArch) was specifically designed to support MVS.

***The notion of a 1MB "segment" is baked into a lot of MVS system and application code. Just like the notion of a 4KB page.
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier22017/01/20 07:48 AM
  Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier22017/01/20 07:48 AM
    Ignore second post (NT)gallier22017/01/20 07:49 AM
  ARM-based supercomputersDavid Kanter2017/01/20 02:55 PM
    ARM-based supercomputersMaynard Handley2017/01/20 08:55 PM
      ARM-based supercomputers lolRobert David Graham2017/01/21 05:34 PM
        ARM-based supercomputers lolnone2017/01/21 06:44 PM
          ARM-based supercomputers lolMaynard Handley2017/01/21 07:36 PM
            ARM-based supercomputers lolMichael S2017/01/22 02:07 AM
            What did you learn?Innocent Bystander2017/01/22 08:26 AM
          ARM-based supercomputers lolanon2017/01/22 02:24 AM
            ARM-based supercomputers lolGabriele Svelto2017/01/23 12:53 AM
              ARM-based supercomputers lolanon2017/01/23 06:40 AM
            ARM-based supercomputers loljuanrga2017/01/29 05:01 AM
              ARM-based supercomputers lolanon2017/01/29 09:35 AM
        ARM-based supercomputers lolBrendan2017/01/21 07:36 PM
         Dumb belief? ROFL (NT)juanrga2017/01/22 02:59 AM
          Dumb belief? ROFLwumpus2017/01/22 11:31 AM
            Dumb belief? ROFLjuanrga2017/01/29 05:13 AM
        ARM-based supercomputers lolDavid Kanter2017/01/22 08:20 AM
    ARM-based supercomputersRichardC2017/01/22 07:54 AM
      ARM-based supercomputersAaron Spink2017/01/22 12:03 PM
        ARM-based supercomputersNiels Jørgen Kruse2017/01/22 12:16 PM
          ARM-based supercomputersAaron Spink2017/01/23 07:01 PM
            ARM-based supercomputersNiels Jørgen Kruse2017/01/23 10:41 PM
              ARM-based supercomputerstarlinian2017/01/24 08:58 AM
                ARM-based supercomputersMichael S2017/01/24 09:05 AM
                  ARM-based supercomputersAaron Spink2017/01/24 10:38 AM
        ARM-based supercomputersdmcq2017/01/22 01:03 PM
          A73 does not have L1 ECCDavid Kanter2017/01/22 09:50 PM
            A73 does not have L1 ECCdmcq2017/01/23 03:32 PM
              A73 does not have L1 ECCMichael S2017/01/23 03:50 PM
              A73 does not have L1 ECCdmcq2017/01/24 08:15 AM
                A73 does not have L1 ECCMichael S2017/01/24 08:47 AM
          ARM SOCs with ECC DRAMMichael S2017/01/23 03:17 AM
          ARM-based supercomputersAaron Spink2017/01/23 07:02 PM
            X-Gene 3 supports 1TB of memoryvvid2017/01/24 01:55 AM
              X-Gene 3 supports 1TB of memoryWilco2017/01/24 02:13 AM
                You seem to be confusing "SoC" with "CPU core" (NT)Anon2017/01/24 03:12 AM
                X-Gene 3 supports 1TB of memoryMichael S2017/01/24 05:26 AM
                  X-Gene 3 supports 1TB of memorydmcq2017/01/24 08:28 AM
                X-Gene 3 supports 1TB of memoryAaron Spink2017/01/24 06:46 AM
                  X-Gene 3 supports 1TB of memoryWilco2017/01/24 03:10 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter2017/01/24 07:31 PM
                X-Gene 3 supports 1TB of memorySimon Farnsworth2017/01/26 02:23 AM
              X-Gene 3 supports 1TB of memoryAaron Spink2017/01/24 06:44 AM
                X-Gene 3 supports 1TB of memoryAnon2017/01/24 01:54 PM
                  X-Gene 3 supports 1TB of memoryWilco2017/01/24 03:07 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter2017/01/24 07:34 PM
                      X-Gene 3 supports 1TB of memorynone2017/01/24 10:48 PM
                        X-Gene 3 supports 1TB of memoryMichael S2017/01/25 01:32 AM
                        X-Gene 3 supports 1TB of memoryDavid Hess2017/01/25 09:47 PM
                          X-Gene 3 supports 1TB of memoryMichael S2017/01/26 12:55 AM
                            Really that different?Daniel B2017/01/26 03:37 AM
                              Really that different?none2017/01/26 05:39 AM
                                Really that different?itsmydamnation2017/01/26 03:55 PM
                                  Really that different?none2017/01/26 11:14 PM
                      X-Gene 3 supports 1TB of memoryWilco2017/01/25 03:22 AM
                        Sharing between servers and phonesDavid Kanter2017/01/25 07:15 AM
                          Sharing between servers and phonesWilco2017/01/25 04:41 PM
                            Sharing between servers and phonesDavid Kanter2017/01/25 06:10 PM
                              Sharing between servers and phonesGabriele Svelto2017/01/26 03:15 AM
                                Sharing between servers and phonesDavid Kanter2017/01/26 07:33 AM
                                  Sharing between servers and phoneswumpus2017/01/26 03:25 PM
                                    Sharing between servers and phonesDavid Kanter2017/01/27 06:46 AM
                                      Any idea why Intel doesn't ship server chips with eDRAM?Mark Roulo2017/01/27 09:02 AM
                                        Any idea why Intel doesn't ship server chips with eDRAM?Per Hesselgren2017/01/28 06:49 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Simon Farnsworth2017/01/28 07:06 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Michael S2017/01/29 02:43 AM
                                            off die eDRAM?wumpus2017/01/29 06:57 AM
                                              off die eDRAM?anon2017/01/29 09:46 AM
                                      Sharing between servers and phonesrwessel2017/01/27 10:36 PM
                                      Sharing between servers and phonesLinus Torvalds2017/01/28 11:49 AM
                                        Sharing between servers and phonesrwessel2017/01/29 08:56 PM
                                          Sharing between servers and phonesLinus Torvalds2017/01/30 10:01 AM
                                            Sharing between servers and phonesrwessel2017/01/31 12:29 AM
                                              Sharing between servers and phonesIreland2017/01/31 07:55 AM
                                                Please keep your posts on topicDavid Kanter2017/01/31 11:32 AM
                                                  Technology that can survive in harsh conditionsIreland2017/01/31 12:51 PM
                                                    Stay on topic, you have been warnedDavid Kanter2017/01/31 06:56 PM
                                                On topic summary and my thoughts on it. Jouni Osmala2017/01/31 01:10 PM
                                                  On topic summary and my thoughts on it. Ireland2017/01/31 01:27 PM
                                              Sharing between servers and phonesLinus Torvalds2017/01/31 01:01 PM
                                                Sharing between servers and phonesLinus Torvalds2017/01/31 01:49 PM
                                                  Sharing between servers and phonesDoug S2017/01/31 02:12 PM
                                                Sharing between servers and phonesrwessel2017/01/31 05:54 PM
                                                  Sharing between servers and phonesLinus Torvalds2017/02/01 09:17 AM
                                                    Sharing between servers and phonesrwessel2017/02/02 03:40 PM
                                                  Sharing between servers and phonesjoncmu2017/02/01 01:36 PM
                                                    Sharing between servers and phonesChristian Borntraeger2017/02/02 03:46 AM
            ARM-based supercomputersRichardC2017/01/24 05:50 AM
              ARM-based supercomputersAaron Spink2017/01/24 07:20 AM
                ARM-based supercomputersdmcq2017/01/24 08:44 AM
                  ARM-based supercomputersnone2017/01/24 09:10 AM
                  ARM-based supercomputersjuanrga2017/01/29 05:33 AM
                    ARM-based supercomputershobel2017/01/30 02:35 AM
                ARM-based supercomputersRichardC2017/01/24 09:34 AM
                  ARM-based supercomputersdmcq2017/01/24 10:10 AM
                  ARM-based supercomputersAaron Spink2017/01/24 11:01 AM
                    ARM-based supercomputersRichardC2017/01/24 04:06 PM
                      ARM-based supercomputersIreland2017/01/24 05:16 PM
                      ARM-based supercomputersAaron Spink2017/01/24 07:43 PM
                    video renderingRichardC2017/01/24 05:08 PM
                      video renderingIreland2017/01/24 05:26 PM
                      video renderingAaron Spink2017/01/24 07:54 PM
                        video renderingRichardC2017/01/25 04:26 AM
                          display bandwidthRichardC2017/01/25 05:30 AM
                          video renderingIreland2017/01/25 08:11 AM
                            You can keep a coal furnace fed all the time. (NT)anon2017/01/25 03:27 PM
                              You can keep a coal furnace fed all the time.Ireland2017/01/25 03:36 PM
                                You can keep a coal furnace fed all the time.anon2017/01/27 03:29 AM
                                  You can keep a coal furnace fed all the time.Michael S2017/01/27 07:22 AM
                                    You can keep a coal furnace fed all the time.Ireland2017/01/27 10:59 AM
                                    You can keep a coal furnace fed all the time.anon2017/01/27 03:09 PM
                                      Information and Super Materials Ireland2017/01/28 11:13 AM
                            video renderingRichardC2017/01/26 11:39 AM
                              video renderingIreland2017/01/26 12:49 PM
                                video renderingIreland2017/01/26 12:58 PM
                                video renderingRichardC2017/01/26 03:24 PM
                                  video renderingIreland2017/01/26 05:09 PM
                          video renderingGabriele Svelto2017/01/25 08:16 AM
                            video renderingIreland2017/01/25 08:33 AM
                              The challenge at Pixar Ireland2017/01/25 08:40 AM
                                Pixar story & technologyRichard Cownie2017/01/25 11:36 AM
                                  Pixar story & technologyIreland2017/01/25 11:58 AM
                          video renderingAaron Spink2017/01/25 02:49 PM
                            video renderingIreland2017/01/25 03:26 PM
                            cloud network infrastructureRichardC2017/01/26 10:47 AM
                              cloud network infrastructureAaron Spink2017/01/26 07:37 PM
                ARM-based supercomputersGabriele Svelto2017/01/24 12:40 PM
                  ARM-based supercomputersAaron Spink2017/01/24 08:00 PM
        ARM-based supercomputersRichardC2017/01/22 09:45 PM
          ARM-based supercomputersGabriele Svelto2017/01/23 01:03 AM
            ARM-based supercomputersRichardC2017/01/23 06:57 AM
      ARM-based supercomputersGabriele Svelto2017/01/23 03:23 AM
    Mont blanc project and ARM HPC in generaljuanrga2017/01/29 04:42 AM
      Juan, why do you have to be such an arrogant twat to the person who provides this forum? (NT)Annoyed2017/01/30 02:32 AM
      Still idioticDavid Kanter2017/01/30 07:49 AM
        Still idioticMr. Camel2017/01/30 08:16 PM
        Still idioticjuanrga2017/02/02 08:11 AM
      Mont blanc project and ARM HPC in generalIreland2017/01/31 04:53 PM
        Four Things to Consider Ireland2017/01/31 05:11 PM
          Four Things to Consider tarlinian2017/01/31 06:38 PM
            Four Things to Consider Ireland2017/01/31 06:58 PM
              no evidence that it goes on a shipRichardC2017/02/01 06:05 AM
                no evidence that it goes on a shipIreland2017/02/02 01:57 PM
                  no evidence that it goes on a shipRichardC2017/02/03 06:04 AM
                    no evidence that it goes on a shipIreland2017/02/03 08:02 AM
              A better place to site a supercomputerAnon2017/02/01 06:57 AM
                A better place to site a supercomputerIreland2017/02/01 06:37 PM
                  A better Ireland..Anon2017/02/01 07:52 PM
                    A better Ireland..slacker2017/02/01 10:32 PM
                    A better Ireland..Ireland2017/02/02 04:06 AM
                    He is manic (NT)anonymo2017/02/02 12:21 PM
                      or a botanon2017/02/02 02:17 PM
                        or a botanonymou52017/02/02 03:46 PM
                        or a botnone2017/02/02 10:45 PM
                          or a botanon2017/02/03 09:30 AM
                        Better bot example: amanfrommars1Doug S2017/02/03 10:44 AM
            Make allowances, the man never sleeps..Anon2017/01/31 08:22 PM
        Mont blanc project and ARM HPC in generaletudiant2017/02/02 08:39 AM
          Mont blanc project and ARM HPC in generalIreland2017/02/02 11:12 AM
            Connection between two different modelsIreland2017/02/02 11:18 AM
              Connection between two different modelsetudiant2017/02/04 02:40 PM
                Connection between two different modelsRichardC2017/02/06 08:36 AM
                  Connection between two different modelsdmcq2017/02/06 10:07 AM
                  Connection between two different modelsIreland2017/02/06 11:17 AM
                    probably not the right forum ...RichardC2017/02/06 12:31 PM
                      probably not the right forum ...Ireland2017/02/06 01:53 PM
Reply to this Topic
Body: No Text
How do you spell green?