Sharing between servers and phones

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), February 1, 2017 10:17 am
Room: Moderated Discussions
rwessel (robertwessel.delete@this.yahoo.com) on January 31, 2017 5:54 pm wrote:
>
> But I'm curious, when is it handy to know which PTE dirtied (or referenced) a page? All
> I'm coming up with is pretty contrived. My best idea is to generate data to decide to
> migrate a page to a better NUMA node (or conversely, manage thread-to-node affinity).
>
> > In fact, a per-virtual-mapping dirty bit is so useful that we end up not just tracking the
> > usual dirty state (that most CPU's give us in hardware), we end up having a second sw-only
> > dirty state that we call the "soft dirty" state for tracking things like "has this page been
> > changed since we last looked at it", which is useful for things like checkpointing.
>
> MVS does the same at the physical page level, for similar reasons. For the reference bit as well.

So I suspect MVS does a lot less physical page sharing.

Unix, with mmap() and fork() ends up often sharing the page with multiple different virtual mappings, and the mappings can also have entirely different protections (ie the same page might be mapped by both root and a normal user).

So one example actually comes from the fairly recent "Dirty COW" security issue: we actually want to check whether this particular mapping already dirtied the page or not. The code did something else exactly because of the s390 difference (ie the dirty bit wasn't actually per-mapping on s390, so the obvious test didn't work there), and that was one of the causes of the security bug.

Other causes tend to be about various TLB shootdown things: if the page table entry is dirty for the mapping that is being torn down, that is different from the page itself being dirty: a dirty page is meaningless from TLB shootdown, but a dirty mapping of a page means that there may still be writing activity going on in that thread, and the TLB needs to be invalidated synchronously (to make sure that the dirty state is in sync with the dirty state of the page) rather than being batched up for later. And batching up TLB invalidates is actually kind of a big deal, so you don't want to do it in general. Ergo: check the dirty bit.

So there are these kinds of details where the per-mapping dirty bit is useful. They're not huge, but when there is one silly unusual architecture that does things differently from all the rest, that can be very annoying when you share code and a lot of logic (and trust me, you absolutely want to share details like page table teardown and TLB shootdown across architectures - it's really really easy to get these things wrong).

So I doubt MVS people or other s390-only users ever cared about the fact that s390 is different. The s390-specific projects could always work around the idiosyncrasies (or even take advantage of them). But a fairly portable project like Linux that support 30 different architectures ends up having these kinds of issues.

> > > But yes, the biggest annoyance was just that we share the VM(code) No disagreement.
>
> Of course, the number of Linux images zArch in use is unlikely
> to drive major internal changes to the kernel.

It turns out that the s390 people made their page tables have a SW-managed dirty bit, and that solved these annoyances.

> BTW, I'm sure your post went wonky after the (code) (where you used angle brackets), probably up to some point
> where he found something else that looked like a tag.

Yeah, very possible. I detest editing html, it's not meant for humans. This site would be better off with markdown or something. Whatever.

> I'm surprised that you can't make use of some of that under the hood for zArch. I assume that somewhere
> there's a "copy space-A/address-B to space-C/address-D for length E" function.

Yes, we might be able to. It's not heavily used, though, because it's a special Linux-only system call that doesn't exist in POSIX, so "not very commonly used" + "s390 isn't that common" = "nobody has ever even looked at it".

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier22017/01/20 08:48 AM
  Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier22017/01/20 08:48 AM
    Ignore second post (NT)gallier22017/01/20 08:49 AM
  ARM-based supercomputersDavid Kanter2017/01/20 03:55 PM
    ARM-based supercomputersMaynard Handley2017/01/20 09:55 PM
      ARM-based supercomputers lolRobert David Graham2017/01/21 06:34 PM
        ARM-based supercomputers lolnone2017/01/21 07:44 PM
          ARM-based supercomputers lolMaynard Handley2017/01/21 08:36 PM
            ARM-based supercomputers lolMichael S2017/01/22 03:07 AM
            What did you learn?Innocent Bystander2017/01/22 09:26 AM
          ARM-based supercomputers lolanon2017/01/22 03:24 AM
            ARM-based supercomputers lolGabriele Svelto2017/01/23 01:53 AM
              ARM-based supercomputers lolanon2017/01/23 07:40 AM
            ARM-based supercomputers loljuanrga2017/01/29 06:01 AM
              ARM-based supercomputers lolanon2017/01/29 10:35 AM
        ARM-based supercomputers lolBrendan2017/01/21 08:36 PM
         Dumb belief? ROFL (NT)juanrga2017/01/22 03:59 AM
          Dumb belief? ROFLwumpus2017/01/22 12:31 PM
            Dumb belief? ROFLjuanrga2017/01/29 06:13 AM
        ARM-based supercomputers lolDavid Kanter2017/01/22 09:20 AM
    ARM-based supercomputersRichardC2017/01/22 08:54 AM
      ARM-based supercomputersAaron Spink2017/01/22 01:03 PM
        ARM-based supercomputersNiels Jørgen Kruse2017/01/22 01:16 PM
          ARM-based supercomputersAaron Spink2017/01/23 08:01 PM
            ARM-based supercomputersNiels Jørgen Kruse2017/01/23 11:41 PM
              ARM-based supercomputerstarlinian2017/01/24 09:58 AM
                ARM-based supercomputersMichael S2017/01/24 10:05 AM
                  ARM-based supercomputersAaron Spink2017/01/24 11:38 AM
        ARM-based supercomputersdmcq2017/01/22 02:03 PM
          A73 does not have L1 ECCDavid Kanter2017/01/22 10:50 PM
            A73 does not have L1 ECCdmcq2017/01/23 04:32 PM
              A73 does not have L1 ECCMichael S2017/01/23 04:50 PM
              A73 does not have L1 ECCdmcq2017/01/24 09:15 AM
                A73 does not have L1 ECCMichael S2017/01/24 09:47 AM
          ARM SOCs with ECC DRAMMichael S2017/01/23 04:17 AM
          ARM-based supercomputersAaron Spink2017/01/23 08:02 PM
            X-Gene 3 supports 1TB of memoryvvid2017/01/24 02:55 AM
              X-Gene 3 supports 1TB of memoryWilco2017/01/24 03:13 AM
                You seem to be confusing "SoC" with "CPU core" (NT)Anon2017/01/24 04:12 AM
                X-Gene 3 supports 1TB of memoryMichael S2017/01/24 06:26 AM
                  X-Gene 3 supports 1TB of memorydmcq2017/01/24 09:28 AM
                X-Gene 3 supports 1TB of memoryAaron Spink2017/01/24 07:46 AM
                  X-Gene 3 supports 1TB of memoryWilco2017/01/24 04:10 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter2017/01/24 08:31 PM
                X-Gene 3 supports 1TB of memorySimon Farnsworth2017/01/26 03:23 AM
              X-Gene 3 supports 1TB of memoryAaron Spink2017/01/24 07:44 AM
                X-Gene 3 supports 1TB of memoryAnon2017/01/24 02:54 PM
                  X-Gene 3 supports 1TB of memoryWilco2017/01/24 04:07 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter2017/01/24 08:34 PM
                      X-Gene 3 supports 1TB of memorynone2017/01/24 11:48 PM
                        X-Gene 3 supports 1TB of memoryMichael S2017/01/25 02:32 AM
                        X-Gene 3 supports 1TB of memoryDavid Hess2017/01/25 10:47 PM
                          X-Gene 3 supports 1TB of memoryMichael S2017/01/26 01:55 AM
                            Really that different?Daniel B2017/01/26 04:37 AM
                              Really that different?none2017/01/26 06:39 AM
                                Really that different?itsmydamnation2017/01/26 04:55 PM
                                  Really that different?none2017/01/27 12:14 AM
                      X-Gene 3 supports 1TB of memoryWilco2017/01/25 04:22 AM
                        Sharing between servers and phonesDavid Kanter2017/01/25 08:15 AM
                          Sharing between servers and phonesWilco2017/01/25 05:41 PM
                            Sharing between servers and phonesDavid Kanter2017/01/25 07:10 PM
                              Sharing between servers and phonesGabriele Svelto2017/01/26 04:15 AM
                                Sharing between servers and phonesDavid Kanter2017/01/26 08:33 AM
                                  Sharing between servers and phoneswumpus2017/01/26 04:25 PM
                                    Sharing between servers and phonesDavid Kanter2017/01/27 07:46 AM
                                      Any idea why Intel doesn't ship server chips with eDRAM?Mark Roulo2017/01/27 10:02 AM
                                        Any idea why Intel doesn't ship server chips with eDRAM?Per Hesselgren2017/01/28 07:49 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Simon Farnsworth2017/01/28 08:06 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Michael S2017/01/29 03:43 AM
                                            off die eDRAM?wumpus2017/01/29 07:57 AM
                                              off die eDRAM?anon2017/01/29 10:46 AM
                                      Sharing between servers and phonesrwessel2017/01/27 11:36 PM
                                      Sharing between servers and phonesLinus Torvalds2017/01/28 12:49 PM
                                        Sharing between servers and phonesrwessel2017/01/29 09:56 PM
                                          Sharing between servers and phonesLinus Torvalds2017/01/30 11:01 AM
                                            Sharing between servers and phonesrwessel2017/01/31 01:29 AM
                                              Sharing between servers and phonesIreland2017/01/31 08:55 AM
                                                Please keep your posts on topicDavid Kanter2017/01/31 12:32 PM
                                                  Technology that can survive in harsh conditionsIreland2017/01/31 01:51 PM
                                                    Stay on topic, you have been warnedDavid Kanter2017/01/31 07:56 PM
                                                On topic summary and my thoughts on it. Jouni Osmala2017/01/31 02:10 PM
                                                  On topic summary and my thoughts on it. Ireland2017/01/31 02:27 PM
                                              Sharing between servers and phonesLinus Torvalds2017/01/31 02:01 PM
                                                Sharing between servers and phonesLinus Torvalds2017/01/31 02:49 PM
                                                  Sharing between servers and phonesDoug S2017/01/31 03:12 PM
                                                Sharing between servers and phonesrwessel2017/01/31 06:54 PM
                                                  Sharing between servers and phonesLinus Torvalds2017/02/01 10:17 AM
                                                    Sharing between servers and phonesrwessel2017/02/02 04:40 PM
                                                  Sharing between servers and phonesjoncmu2017/02/01 02:36 PM
                                                    Sharing between servers and phonesChristian Borntraeger2017/02/02 04:46 AM
            ARM-based supercomputersRichardC2017/01/24 06:50 AM
              ARM-based supercomputersAaron Spink2017/01/24 08:20 AM
                ARM-based supercomputersdmcq2017/01/24 09:44 AM
                  ARM-based supercomputersnone2017/01/24 10:10 AM
                  ARM-based supercomputersjuanrga2017/01/29 06:33 AM
                    ARM-based supercomputershobel2017/01/30 03:35 AM
                ARM-based supercomputersRichardC2017/01/24 10:34 AM
                  ARM-based supercomputersdmcq2017/01/24 11:10 AM
                  ARM-based supercomputersAaron Spink2017/01/24 12:01 PM
                    ARM-based supercomputersRichardC2017/01/24 05:06 PM
                      ARM-based supercomputersIreland2017/01/24 06:16 PM
                      ARM-based supercomputersAaron Spink2017/01/24 08:43 PM
                    video renderingRichardC2017/01/24 06:08 PM
                      video renderingIreland2017/01/24 06:26 PM
                      video renderingAaron Spink2017/01/24 08:54 PM
                        video renderingRichardC2017/01/25 05:26 AM
                          display bandwidthRichardC2017/01/25 06:30 AM
                          video renderingIreland2017/01/25 09:11 AM
                            You can keep a coal furnace fed all the time. (NT)anon2017/01/25 04:27 PM
                              You can keep a coal furnace fed all the time.Ireland2017/01/25 04:36 PM
                                You can keep a coal furnace fed all the time.anon2017/01/27 04:29 AM
                                  You can keep a coal furnace fed all the time.Michael S2017/01/27 08:22 AM
                                    You can keep a coal furnace fed all the time.Ireland2017/01/27 11:59 AM
                                    You can keep a coal furnace fed all the time.anon2017/01/27 04:09 PM
                                      Information and Super Materials Ireland2017/01/28 12:13 PM
                            video renderingRichardC2017/01/26 12:39 PM
                              video renderingIreland2017/01/26 01:49 PM
                                video renderingIreland2017/01/26 01:58 PM
                                video renderingRichardC2017/01/26 04:24 PM
                                  video renderingIreland2017/01/26 06:09 PM
                          video renderingGabriele Svelto2017/01/25 09:16 AM
                            video renderingIreland2017/01/25 09:33 AM
                              The challenge at Pixar Ireland2017/01/25 09:40 AM
                                Pixar story & technologyRichard Cownie2017/01/25 12:36 PM
                                  Pixar story & technologyIreland2017/01/25 12:58 PM
                          video renderingAaron Spink2017/01/25 03:49 PM
                            video renderingIreland2017/01/25 04:26 PM
                            cloud network infrastructureRichardC2017/01/26 11:47 AM
                              cloud network infrastructureAaron Spink2017/01/26 08:37 PM
                ARM-based supercomputersGabriele Svelto2017/01/24 01:40 PM
                  ARM-based supercomputersAaron Spink2017/01/24 09:00 PM
        ARM-based supercomputersRichardC2017/01/22 10:45 PM
          ARM-based supercomputersGabriele Svelto2017/01/23 02:03 AM
            ARM-based supercomputersRichardC2017/01/23 07:57 AM
      ARM-based supercomputersGabriele Svelto2017/01/23 04:23 AM
    Mont blanc project and ARM HPC in generaljuanrga2017/01/29 05:42 AM
      Juan, why do you have to be such an arrogant twat to the person who provides this forum? (NT)Annoyed2017/01/30 03:32 AM
      Still idioticDavid Kanter2017/01/30 08:49 AM
        Still idioticMr. Camel2017/01/30 09:16 PM
        Still idioticjuanrga2017/02/02 09:11 AM
      Mont blanc project and ARM HPC in generalIreland2017/01/31 05:53 PM
        Four Things to Consider Ireland2017/01/31 06:11 PM
          Four Things to Consider tarlinian2017/01/31 07:38 PM
            Four Things to Consider Ireland2017/01/31 07:58 PM
              no evidence that it goes on a shipRichardC2017/02/01 07:05 AM
                no evidence that it goes on a shipIreland2017/02/02 02:57 PM
                  no evidence that it goes on a shipRichardC2017/02/03 07:04 AM
                    no evidence that it goes on a shipIreland2017/02/03 09:02 AM
              A better place to site a supercomputerAnon2017/02/01 07:57 AM
                A better place to site a supercomputerIreland2017/02/01 07:37 PM
                  A better Ireland..Anon2017/02/01 08:52 PM
                    A better Ireland..slacker2017/02/01 11:32 PM
                    A better Ireland..Ireland2017/02/02 05:06 AM
                    He is manic (NT)anonymo2017/02/02 01:21 PM
                      or a botanon2017/02/02 03:17 PM
                        or a botanonymou52017/02/02 04:46 PM
                        or a botnone2017/02/02 11:45 PM
                          or a botanon2017/02/03 10:30 AM
                        Better bot example: amanfrommars1Doug S2017/02/03 11:44 AM
            Make allowances, the man never sleeps..Anon2017/01/31 09:22 PM
        Mont blanc project and ARM HPC in generaletudiant2017/02/02 09:39 AM
          Mont blanc project and ARM HPC in generalIreland2017/02/02 12:12 PM
            Connection between two different modelsIreland2017/02/02 12:18 PM
              Connection between two different modelsetudiant2017/02/04 03:40 PM
                Connection between two different modelsRichardC2017/02/06 09:36 AM
                  Connection between two different modelsdmcq2017/02/06 11:07 AM
                  Connection between two different modelsIreland2017/02/06 12:17 PM
                    probably not the right forum ...RichardC2017/02/06 01:31 PM
                      probably not the right forum ...Ireland2017/02/06 02:53 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?