Sharing between servers and phones

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), February 1, 2017 10:17 am
Room: Moderated Discussions
rwessel (robertwessel.delete@this.yahoo.com) on January 31, 2017 5:54 pm wrote:
>
> But I'm curious, when is it handy to know which PTE dirtied (or referenced) a page? All
> I'm coming up with is pretty contrived. My best idea is to generate data to decide to
> migrate a page to a better NUMA node (or conversely, manage thread-to-node affinity).
>
> > In fact, a per-virtual-mapping dirty bit is so useful that we end up not just tracking the
> > usual dirty state (that most CPU's give us in hardware), we end up having a second sw-only
> > dirty state that we call the "soft dirty" state for tracking things like "has this page been
> > changed since we last looked at it", which is useful for things like checkpointing.
>
> MVS does the same at the physical page level, for similar reasons. For the reference bit as well.

So I suspect MVS does a lot less physical page sharing.

Unix, with mmap() and fork() ends up often sharing the page with multiple different virtual mappings, and the mappings can also have entirely different protections (ie the same page might be mapped by both root and a normal user).

So one example actually comes from the fairly recent "Dirty COW" security issue: we actually want to check whether this particular mapping already dirtied the page or not. The code did something else exactly because of the s390 difference (ie the dirty bit wasn't actually per-mapping on s390, so the obvious test didn't work there), and that was one of the causes of the security bug.

Other causes tend to be about various TLB shootdown things: if the page table entry is dirty for the mapping that is being torn down, that is different from the page itself being dirty: a dirty page is meaningless from TLB shootdown, but a dirty mapping of a page means that there may still be writing activity going on in that thread, and the TLB needs to be invalidated synchronously (to make sure that the dirty state is in sync with the dirty state of the page) rather than being batched up for later. And batching up TLB invalidates is actually kind of a big deal, so you don't want to do it in general. Ergo: check the dirty bit.

So there are these kinds of details where the per-mapping dirty bit is useful. They're not huge, but when there is one silly unusual architecture that does things differently from all the rest, that can be very annoying when you share code and a lot of logic (and trust me, you absolutely want to share details like page table teardown and TLB shootdown across architectures - it's really really easy to get these things wrong).

So I doubt MVS people or other s390-only users ever cared about the fact that s390 is different. The s390-specific projects could always work around the idiosyncrasies (or even take advantage of them). But a fairly portable project like Linux that support 30 different architectures ends up having these kinds of issues.

> > > But yes, the biggest annoyance was just that we share the VM(code) No disagreement.
>
> Of course, the number of Linux images zArch in use is unlikely
> to drive major internal changes to the kernel.

It turns out that the s390 people made their page tables have a SW-managed dirty bit, and that solved these annoyances.

> BTW, I'm sure your post went wonky after the (code) (where you used angle brackets), probably up to some point
> where he found something else that looked like a tag.

Yeah, very possible. I detest editing html, it's not meant for humans. This site would be better off with markdown or something. Whatever.

> I'm surprised that you can't make use of some of that under the hood for zArch. I assume that somewhere
> there's a "copy space-A/address-B to space-C/address-D for length E" function.

Yes, we might be able to. It's not heavily used, though, because it's a special Linux-only system call that doesn't exist in POSIX, so "not very commonly used" + "s390 isn't that common" = "nobody has ever even looked at it".

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier201/20/17 08:48 AM
  Cray to Develop ARM-based Isambard Supercomputer for UK Met Officegallier201/20/17 08:48 AM
    Ignore second post (NT)gallier201/20/17 08:49 AM
  ARM-based supercomputersDavid Kanter01/20/17 03:55 PM
    ARM-based supercomputersMaynard Handley01/20/17 09:55 PM
      ARM-based supercomputers lolRobert David Graham01/21/17 06:34 PM
        ARM-based supercomputers lolnone01/21/17 07:44 PM
          ARM-based supercomputers lolMaynard Handley01/21/17 08:36 PM
            ARM-based supercomputers lolMichael S01/22/17 03:07 AM
            What did you learn?Innocent Bystander01/22/17 09:26 AM
          ARM-based supercomputers lolanon01/22/17 03:24 AM
            ARM-based supercomputers lolGabriele Svelto01/23/17 01:53 AM
              ARM-based supercomputers lolanon01/23/17 07:40 AM
            ARM-based supercomputers loljuanrga01/29/17 06:01 AM
              ARM-based supercomputers lolanon01/29/17 10:35 AM
        ARM-based supercomputers lolBrendan01/21/17 08:36 PM
         Dumb belief? ROFL (NT)juanrga01/22/17 03:59 AM
          Dumb belief? ROFLwumpus01/22/17 12:31 PM
            Dumb belief? ROFLjuanrga01/29/17 06:13 AM
        ARM-based supercomputers lolDavid Kanter01/22/17 09:20 AM
    ARM-based supercomputersRichardC01/22/17 08:54 AM
      ARM-based supercomputersAaron Spink01/22/17 01:03 PM
        ARM-based supercomputersNiels Jørgen Kruse01/22/17 01:16 PM
          ARM-based supercomputersAaron Spink01/23/17 08:01 PM
            ARM-based supercomputersNiels Jørgen Kruse01/23/17 11:41 PM
              ARM-based supercomputerstarlinian01/24/17 09:58 AM
                ARM-based supercomputersMichael S01/24/17 10:05 AM
                  ARM-based supercomputersAaron Spink01/24/17 11:38 AM
        ARM-based supercomputersdmcq01/22/17 02:03 PM
          A73 does not have L1 ECCDavid Kanter01/22/17 10:50 PM
            A73 does not have L1 ECCdmcq01/23/17 04:32 PM
              A73 does not have L1 ECCMichael S01/23/17 04:50 PM
              A73 does not have L1 ECCdmcq01/24/17 09:15 AM
                A73 does not have L1 ECCMichael S01/24/17 09:47 AM
          ARM SOCs with ECC DRAMMichael S01/23/17 04:17 AM
          ARM-based supercomputersAaron Spink01/23/17 08:02 PM
            X-Gene 3 supports 1TB of memoryvvid01/24/17 02:55 AM
              X-Gene 3 supports 1TB of memoryWilco01/24/17 03:13 AM
                You seem to be confusing "SoC" with "CPU core" (NT)Anon01/24/17 04:12 AM
                X-Gene 3 supports 1TB of memoryMichael S01/24/17 06:26 AM
                  X-Gene 3 supports 1TB of memorydmcq01/24/17 09:28 AM
                X-Gene 3 supports 1TB of memoryAaron Spink01/24/17 07:46 AM
                  X-Gene 3 supports 1TB of memoryWilco01/24/17 04:10 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter01/24/17 08:31 PM
                X-Gene 3 supports 1TB of memorySimon Farnsworth01/26/17 03:23 AM
              X-Gene 3 supports 1TB of memoryAaron Spink01/24/17 07:44 AM
                X-Gene 3 supports 1TB of memoryAnon01/24/17 02:54 PM
                  X-Gene 3 supports 1TB of memoryWilco01/24/17 04:07 PM
                    X-Gene 3 supports 1TB of memoryDavid Kanter01/24/17 08:34 PM
                      X-Gene 3 supports 1TB of memorynone01/24/17 11:48 PM
                        X-Gene 3 supports 1TB of memoryMichael S01/25/17 02:32 AM
                        X-Gene 3 supports 1TB of memoryDavid Hess01/25/17 10:47 PM
                          X-Gene 3 supports 1TB of memoryMichael S01/26/17 01:55 AM
                            Really that different?Daniel B01/26/17 04:37 AM
                              Really that different?none01/26/17 06:39 AM
                                Really that different?itsmydamnation01/26/17 04:55 PM
                                  Really that different?none01/27/17 12:14 AM
                      X-Gene 3 supports 1TB of memoryWilco01/25/17 04:22 AM
                        Sharing between servers and phonesDavid Kanter01/25/17 08:15 AM
                          Sharing between servers and phonesWilco01/25/17 05:41 PM
                            Sharing between servers and phonesDavid Kanter01/25/17 07:10 PM
                              Sharing between servers and phonesGabriele Svelto01/26/17 04:15 AM
                                Sharing between servers and phonesDavid Kanter01/26/17 08:33 AM
                                  Sharing between servers and phoneswumpus01/26/17 04:25 PM
                                    Sharing between servers and phonesDavid Kanter01/27/17 07:46 AM
                                      Any idea why Intel doesn't ship server chips with eDRAM?Mark Roulo01/27/17 10:02 AM
                                        Any idea why Intel doesn't ship server chips with eDRAM?Per Hesselgren01/28/17 07:49 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Simon Farnsworth01/28/17 08:06 AM
                                          Any idea why Intel doesn't ship server chips with eDRAM?Michael S01/29/17 03:43 AM
                                            off die eDRAM?wumpus01/29/17 07:57 AM
                                              off die eDRAM?anon01/29/17 10:46 AM
                                      Sharing between servers and phonesrwessel01/27/17 11:36 PM
                                      Sharing between servers and phonesLinus Torvalds01/28/17 12:49 PM
                                        Sharing between servers and phonesrwessel01/29/17 09:56 PM
                                          Sharing between servers and phonesLinus Torvalds01/30/17 11:01 AM
                                            Sharing between servers and phonesrwessel01/31/17 01:29 AM
                                              Sharing between servers and phonesIreland01/31/17 08:55 AM
                                                Please keep your posts on topicDavid Kanter01/31/17 12:32 PM
                                                  Technology that can survive in harsh conditionsIreland01/31/17 01:51 PM
                                                    Stay on topic, you have been warnedDavid Kanter01/31/17 07:56 PM
                                                On topic summary and my thoughts on it. Jouni Osmala01/31/17 02:10 PM
                                                  On topic summary and my thoughts on it. Ireland01/31/17 02:27 PM
                                              Sharing between servers and phonesLinus Torvalds01/31/17 02:01 PM
                                                Sharing between servers and phonesLinus Torvalds01/31/17 02:49 PM
                                                  Sharing between servers and phonesDoug S01/31/17 03:12 PM
                                                Sharing between servers and phonesrwessel01/31/17 06:54 PM
                                                  Sharing between servers and phonesLinus Torvalds02/01/17 10:17 AM
                                                    Sharing between servers and phonesrwessel02/02/17 04:40 PM
                                                  Sharing between servers and phonesjoncmu02/01/17 02:36 PM
                                                    Sharing between servers and phonesChristian Borntraeger02/02/17 04:46 AM
            ARM-based supercomputersRichardC01/24/17 06:50 AM
              ARM-based supercomputersAaron Spink01/24/17 08:20 AM
                ARM-based supercomputersdmcq01/24/17 09:44 AM
                  ARM-based supercomputersnone01/24/17 10:10 AM
                  ARM-based supercomputersjuanrga01/29/17 06:33 AM
                    ARM-based supercomputershobel01/30/17 03:35 AM
                ARM-based supercomputersRichardC 01/24/17 10:34 AM
                  ARM-based supercomputersdmcq01/24/17 11:10 AM
                  ARM-based supercomputersAaron Spink01/24/17 12:01 PM
                    ARM-based supercomputersRichardC01/24/17 05:06 PM
                      ARM-based supercomputersIreland01/24/17 06:16 PM
                      ARM-based supercomputersAaron Spink01/24/17 08:43 PM
                    video renderingRichardC01/24/17 06:08 PM
                      video renderingIreland01/24/17 06:26 PM
                      video renderingAaron Spink01/24/17 08:54 PM
                        video renderingRichardC01/25/17 05:26 AM
                          display bandwidthRichardC01/25/17 06:30 AM
                          video renderingIreland01/25/17 09:11 AM
                            You can keep a coal furnace fed all the time. (NT)anon01/25/17 04:27 PM
                              You can keep a coal furnace fed all the time.Ireland01/25/17 04:36 PM
                                You can keep a coal furnace fed all the time.anon01/27/17 04:29 AM
                                  You can keep a coal furnace fed all the time.Michael S01/27/17 08:22 AM
                                    You can keep a coal furnace fed all the time.Ireland01/27/17 11:59 AM
                                    You can keep a coal furnace fed all the time.anon01/27/17 04:09 PM
                                      Information and Super Materials Ireland01/28/17 12:13 PM
                            video renderingRichardC01/26/17 12:39 PM
                              video renderingIreland01/26/17 01:49 PM
                                video renderingIreland01/26/17 01:58 PM
                                video renderingRichardC01/26/17 04:24 PM
                                  video renderingIreland01/26/17 06:09 PM
                          video renderingGabriele Svelto01/25/17 09:16 AM
                            video renderingIreland01/25/17 09:33 AM
                              The challenge at Pixar Ireland01/25/17 09:40 AM
                                Pixar story & technologyRichard Cownie01/25/17 12:36 PM
                                  Pixar story & technologyIreland01/25/17 12:58 PM
                          video renderingAaron Spink01/25/17 03:49 PM
                            video renderingIreland01/25/17 04:26 PM
                            cloud network infrastructureRichardC01/26/17 11:47 AM
                              cloud network infrastructureAaron Spink01/26/17 08:37 PM
                ARM-based supercomputersGabriele Svelto01/24/17 01:40 PM
                  ARM-based supercomputersAaron Spink01/24/17 09:00 PM
        ARM-based supercomputersRichardC01/22/17 10:45 PM
          ARM-based supercomputersGabriele Svelto01/23/17 02:03 AM
            ARM-based supercomputersRichardC01/23/17 07:57 AM
      ARM-based supercomputersGabriele Svelto01/23/17 04:23 AM
    Mont blanc project and ARM HPC in generaljuanrga01/29/17 05:42 AM
      Juan, why do you have to be such an arrogant twat to the person who provides this forum? (NT)Annoyed01/30/17 03:32 AM
      Still idioticDavid Kanter01/30/17 08:49 AM
        Still idioticMr. Camel01/30/17 09:16 PM
        Still idioticjuanrga02/02/17 09:11 AM
      Mont blanc project and ARM HPC in generalIreland01/31/17 05:53 PM
        Four Things to Consider Ireland01/31/17 06:11 PM
          Four Things to Consider tarlinian01/31/17 07:38 PM
            Four Things to Consider Ireland01/31/17 07:58 PM
              no evidence that it goes on a shipRichardC02/01/17 07:05 AM
                no evidence that it goes on a shipIreland02/02/17 02:57 PM
                  no evidence that it goes on a shipRichardC02/03/17 07:04 AM
                    no evidence that it goes on a shipIreland02/03/17 09:02 AM
              A better place to site a supercomputerAnon02/01/17 07:57 AM
                A better place to site a supercomputerIreland02/01/17 07:37 PM
                  A better Ireland..Anon02/01/17 08:52 PM
                    A better Ireland..slacker02/01/17 11:32 PM
                    A better Ireland..Ireland02/02/17 05:06 AM
                    He is manic (NT)anonymo02/02/17 01:21 PM
                      or a botanon02/02/17 03:17 PM
                        or a botanonymou502/02/17 04:46 PM
                        or a botnone02/02/17 11:45 PM
                          or a botanon02/03/17 10:30 AM
                        Better bot example: amanfrommars1Doug S02/03/17 11:44 AM
            Make allowances, the man never sleeps..Anon01/31/17 09:22 PM
        Mont blanc project and ARM HPC in generaletudiant02/02/17 09:39 AM
          Mont blanc project and ARM HPC in generalIreland02/02/17 12:12 PM
            Connection between two different modelsIreland02/02/17 12:18 PM
              Connection between two different modelsetudiant02/04/17 03:40 PM
                Connection between two different modelsRichardC02/06/17 09:36 AM
                  Connection between two different modelsdmcq02/06/17 11:07 AM
                  Connection between two different modelsIreland02/06/17 12:17 PM
                    probably not the right forum ...RichardC02/06/17 01:31 PM
                      probably not the right forum ...Ireland02/06/17 02:53 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?