Example of memory ordering causing excitement in the real world

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), July 15, 2015 6:35 pm
Room: Moderated Discussions
Wilco (Wilco.Dijkstra.delete@this.ntlworld.com) on July 15, 2015 2:43 pm wrote:
>
> So is this reordering causal according to you? I bet most people don't find it intuitive
> either as this kind of reordering is exactly what is expected on weak memory models.

So that re-ordering is indeed causal, although it is not sequentially consistent.

There is absolutely no question that

(a) sequential consistency is the easiest one to understand (basically: "everything is ordered")

(b) x86 is famously strongly ordered, but it is not that strongly ordered.

A causal consistency involves a causal relationship: "if A then B". That means you have a read of a variable followed by a write to (possibly another) variable. The read is the conditional, the write is the action.

It's called "causal" since the read of the first variable could be the cause of the write to the second one. That cause can be in many ways - it could affect whether the write is done at all, it could affect the actual value written, and it could affect which variable to be written.

So a causal chain would be something like

IF (A) THEN B=1

and causal consistency means that if another CPU now sees "B" having the value 1, then it had better see that A was true too. In memory ordering documentations you would skip the "if/then" part and only talk about the actual accesses, so you have a pattern of

read A
B=1

to show that causal relationship, and then the rule would be that another CPU that does

read B
read A

would never see "B and !A" (because B having the value 1 is caused by A having a non-zero value).

In contrast, your example is about sequential consistency, not causal consistency: when you have the reverse pattern

A=1
read B

(and on another CPU, doing the same thing, except switching the variables: "B=1" followed by "read A") clearly neither CPU had the actions they did be "caused" by anything.

They didn't read anything from memory that could affect the end result: they did an unconditional store, and then a unconditional read. So there is no "if A then B" causal relationship - they are sequential, but not causal.

So "sequential consistency" means that everybody agrees about sequential ordering. "Causal consistency" is weaker than that, and means that just everybody agrees about causal ordering.

Again, I do think that sequential memory consistency is by far the easiest one to understand, and the one that would result in the fewest bugs. If I had the choice, as a software person I'd pick sequential consistency. No argument about it.

And again, x86 is absolutely not sequentially consistent. Never has been. The write buffering means very fundamentally that stores will be delayed past later reads, and that thus stores are clearly not sequentially ordered with reads. And it makes it very easy to see that behavior that you see in that example.

So I've never claimed that x86 was perfect. All I've claimed is that the x86 memory ordering is better. It's not "best".

Engineering side note: I also do claim that from a pure engineering standpoint, the whole "buffer stores" model is at least a reasonable trade-off. Truly sequentially consistent memory ordering is really really hard to make go fast. It's simply not practical. So while I as a software person would find sequential consistency much easier to think about, and never have to worry about memory re-ordering at all, I do see the huge advantage you get from having a store buffer.

In contrast, the advantages of weakening the memory model further are much less obvious. Stores really have to be delayed for good performance. No other memory re-ordering matters nearly as much. Re-ordering loads past loads? Not nearly as big a deal. Just move the first load even earlier. Re-ordering stores past stores? No biggie. They're buffered and not critical anyway, so from a performance standpoint it's pretty much a no-op in the big picture.

(And yes, that's certainly simplifying a bit. A ordered store buffer might need to be bigger than an unordered one, and might not as easily catch some merging behavior. Similarly, re-ordering reads past other reads might make for smaller queues. But those are second-order effects at best, not at all the kind of fundamental issue that is "buffer stores").

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cyclone Macro-op fusionMaynard Handley2015/07/05 06:01 PM
  Cyclone Macro-op fusionGabriele Svelto2015/07/06 02:11 AM
    Cyclone Macro-op fusionMaynard Handley2015/07/06 10:25 AM
      Cyclone Macro-op fusionWilco2015/07/06 11:17 AM
        Cyclone Macro-op fusionMaynard Handley2015/07/06 12:07 PM
          Cyclone Macro-op fusionMaynard Handley2015/07/06 12:55 PM
            Cyclone Macro-op fusionGabriele Svelto2015/07/07 02:49 AM
      Power8 branch conversionSHK2015/07/06 12:41 PM
        Power8 branch conversionMaynard Handley2015/07/06 01:01 PM
          Power8 branch conversionMaynard Handley2015/07/06 01:09 PM
        Benefit of dynamic hammock predicationPaul A. Clayton2015/07/06 03:02 PM
          Benefit of dynamic hammock predicationMaynard Handley2015/07/06 03:34 PM
            Benefit of dynamic hammock predicationEduardoS2015/07/06 07:56 PM
            Benefit of dynamic hammock predicationPatrick Chase2015/07/06 10:30 PM
          Benefit of dynamic hammock predicationLinus Torvalds2015/07/06 03:59 PM
            Benefit of dynamic hammock predicationWilco2015/07/06 04:35 PM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/06 05:25 PM
                Benefit of dynamic hammock predicationanon2015/07/06 09:02 PM
            Benefit of dynamic hammock predicationPatrick Chase2015/07/06 10:31 PM
              Benefit of dynamic hammock predicationanon2015/07/06 11:02 PM
              Benefit of dynamic hammock predicationMichael S2015/07/07 01:07 AM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/07 09:23 AM
                Benefit of dynamic hammock predicationPatrick Chase2015/07/07 10:49 AM
                  Benefit of dynamic hammock predicationSHK2015/07/07 11:34 AM
                  Benefit of dynamic hammock predicationMaynard Handley2015/07/07 12:00 PM
                    Benefit of dynamic hammock predicationExophase2015/07/07 01:26 PM
                      Benefit of dynamic hammock predicationWilco2015/07/07 04:54 PM
                      Benefit of dynamic hammock predicationMaynard Handley2015/07/07 05:25 PM
                    Benefit of dynamic hammock predicationPatrick Chase2015/07/07 02:32 PM
                    Benefit of dynamic hammock predicationMichael S2015/07/08 01:57 AM
                  Benefit of dynamic hammock predicationSylvain Collange2015/07/08 06:30 AM
                    Benefit of dynamic hammock predicationMaynard Handley2015/07/08 09:46 AM
                      Benefit of dynamic hammock predicationSylvain Collange2015/07/08 10:32 AM
                        Benefit of dynamic hammock predicationMaynard Handley2015/07/08 10:57 AM
                        Sadly, skewed associativity has not been broadly adoptedPaul A. Clayton2015/07/08 02:00 PM
                        Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/08 08:23 PM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)David Kanter2015/07/09 07:20 AM
                            Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/09 10:04 AM
                              Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Tim McCaffrey2015/07/09 10:41 AM
                                Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 01:08 PM
                                  Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Sylvain Collange2015/07/10 02:03 AM
                              Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)David Kanter2015/07/09 04:15 PM
                                Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/09 05:21 PM
                            Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 12:44 PM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 09:43 AM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)someone2015/07/10 05:43 AM
            Benefit of dynamic hammock predicationKonrad Schwarz2015/07/07 07:54 AM
            Benefit of dynamic hammock predicationdmcq2015/07/07 02:38 PM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/08 08:23 AM
                Benefit of dynamic hammock predicationdmcq2015/07/08 10:11 AM
                  Benefit of dynamic hammock predicationLinus Torvalds2015/07/08 01:38 PM
                    Example of memory ordering causing excitement in the real worldMark Roulo2015/07/08 04:54 PM
                      Example of memory ordering causing excitement in the real worlddmcq2015/07/09 04:37 AM
                        Example of memory ordering causing excitement in the real worldMark Roulo2015/07/09 08:24 AM
                          Example of memory ordering causing excitement in the real worlddmcq2015/07/09 10:09 AM
                          Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/09 10:12 AM
                            Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/09 12:14 PM
                              Example of memory ordering causing excitement in the real worlddmcq2015/07/09 04:24 PM
                                Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/10 01:07 PM
                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/10 03:49 PM
                                    Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/12 01:02 PM
                                      Example of memory ordering causing excitement in the real worlddmcq2015/07/13 05:03 AM
                                        Example of memory ordering causing excitement in the real worldMichael S2015/07/13 05:19 AM
                                        Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/14 01:21 PM
                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/14 02:54 PM
                                            Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/15 01:13 PM
                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/15 03:43 PM
                                                Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/16 12:41 PM
                                                  Example of memory ordering causing excitement in the real worldWilco2015/07/17 01:57 PM
                                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/17 04:29 PM
                                                      Example of memory ordering causing excitement in the real worlddmcq2015/07/18 04:07 AM
                                          Example of memory ordering causing excitement in the real worldJouni Osmala2015/07/16 03:52 AM
                                            Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/16 01:05 PM
                                              Example of memory ordering causing excitement in the real worldJouni Osmala2015/07/17 01:18 AM
                                                ISA support for bound checkingMichael S2015/07/17 04:51 AM
                                                  ISA support for bound checkingEtienne2015/07/17 06:39 AM
                                                  MPX on SkylakeMark Roulo2015/07/17 08:19 AM
                                                    MPX on SkylakeMichael S2015/07/17 08:44 AM
                                                      MPX on SkylakeMark Roulo2015/07/17 09:03 AM
                                  Example of memory ordering causing excitement in the real worldanon2015/07/12 03:42 AM
                                    Example of memory ordering causing excitement in the real worlddmcq2015/07/12 10:27 AM
                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 11:24 AM
                                      Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 11:53 AM
                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 01:07 PM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/12 07:52 PM
                                            Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/13 11:05 AM
                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/13 12:20 PM
                                                Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/13 01:46 PM
                                                  Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/13 02:10 PM
                                                    Example of memory ordering causing excitement in the real worldanon2015/07/13 08:51 PM
                                                    Simple cores justifying weak ordering?Paul A. Clayton2015/07/14 09:01 AM
                                                      David's Paul A. Clayton2015/07/14 09:37 AM
                                                      David's 2015-07-12 10:34 post confirmed some of that (NT)Paul A. Clayton2015/07/14 09:39 AM
                                                    Example of memory ordering causing excitement in the real worldEtienne2015/07/17 01:15 AM
                                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/13 02:14 PM
                                                  Reality is even worseTerry Gray2015/07/13 06:05 PM
                                                    Reality is even worseKonrad Schwarz2015/07/21 12:13 AM
                                                  A simple CAS/MCAS semantics2015/07/15 12:54 PM
                                                    A simple CAS/MCAS semanticsLinus Torvalds2015/07/15 01:55 PM
                                                    A simple CAS/MCAS semanticsEduardoS2015/07/15 02:31 PM
                                                Example of memory ordering causing excitement in the real worldsylt2015/07/13 02:19 PM
                                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/13 03:24 PM
                                                    Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 04:48 PM
                                                      Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/13 07:18 PM
                                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 07:57 PM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/13 09:01 PM
                                                    What high level languages do you have in mind?Mark Roulo2015/07/13 05:00 PM
                                                      What high level languages do you have in mind?dmcq2015/07/14 06:54 AM
                                                  Example of memory ordering causing excitement in the real worldWilco2015/07/13 04:36 PM
                                                    Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 04:56 PM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/13 06:43 PM
                                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 08:11 PM
                                                          Example of memory ordering causing excitement in the real worldanon2015/07/13 09:07 PM
                                      Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/12 12:32 PM
                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 01:35 PM
                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 01:38 PM
                                          Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 06:47 PM
                                            Example of memory ordering causing excitement in the real worldNoSpammer2015/07/13 12:34 PM
                                              Example of memory ordering causing excitement in the real worldWilco2015/07/13 04:50 PM
                                                Example of memory ordering causing excitement in the real worldNoSpammer2015/07/14 02:02 PM
                                                  Example of memory ordering causing excitement in the real worldanon2015/07/14 09:37 PM
                                                    Example of memory ordering causing excitement in the real worldMichael S2015/07/15 01:19 AM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/15 02:26 AM
                                                        Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 07:05 AM
                                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/15 08:04 AM
                                                            Example of memory ordering causing excitement in the real worldMichael S2015/07/15 08:29 AM
                                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/15 10:09 AM
                                                              Can string ops use microarchitectural state?Paul A. Clayton2015/07/16 09:33 AM
                                                                Can string ops use microarchitectural state?rwessel2015/07/16 01:00 PM
                                                                  Can string ops use microarchitectural state?Paul A. Clayton2015/07/16 06:16 PM
                                                                    Can string ops use microarchitectural state?rwessel2015/07/16 09:34 PM
                                                                      Can string ops use microarchitectural state?Michael S2015/07/17 05:39 AM
                                                                        Can string ops use microarchitectural state?rwessel2015/07/17 09:02 AM
                                                                Consider interrupt followed by task preemptionMichael S2015/07/16 04:31 PM
                                                                  Consider interrupt followed by task preemptionPaul A. Clayton2015/07/16 06:23 PM
                                                            Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 12:50 PM
                                                              Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 02:27 PM
                                                                Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 03:14 PM
                                                                  Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 05:42 PM
                                                      Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 02:44 AM
                                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 02:17 PM
                                                      Example of memory ordering causing excitement in the real worldWilco2015/07/15 02:43 PM
                                                        Example of memory ordering causing excitement in the real worldanon2015/07/15 05:45 PM
                                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 06:35 PM
                                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/16 02:46 AM
                                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/16 02:49 AM
                                        Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 07:15 AM
                                      Example of memory ordering causing excitement in the real worldanon2015/07/12 07:42 PM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/12 09:48 PM
                                        Example of memory ordering causing excitement in the real worldDavid Kanter2015/07/12 10:34 PM
                                          Example of memory ordering causing excitement in the real worldMichael S2015/07/13 01:51 AM
                                            Example of memory ordering causing excitement in the real worldDavid Kanter2015/07/13 08:31 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/13 06:05 AM
                                            Example of memory ordering causing excitement in the real worldGabriele Svelto2015/07/17 05:40 AM
                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/13 08:09 AM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/13 07:46 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/14 04:12 AM
                                        Example of memory ordering causing excitement in the real worlddmcq2015/07/15 03:51 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/15 04:38 AM
                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/15 06:21 AM
                                      Store queues in POWER/PowerPC processorsGabriele Svelto2015/07/16 01:55 AM
                                        Not sure about that conclusionDavid Kanter2015/07/16 07:34 AM
                                          The POWER4 had the same structureGabriele Svelto2015/07/17 05:20 AM
                                        Committing stores in program orderPaul A. Clayton2015/07/16 01:35 PM
                                          Committing stores in program orderMaynard Handley2015/07/17 12:52 PM
                                      Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/20 04:44 AM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/20 07:29 AM
                                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 12:08 AM
                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/21 01:27 AM
                                            Example of memory ordering causing excitement in the real worldMichael S2015/07/21 03:11 AM
                                              Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 05:24 AM
                                            Example of memory ordering causing excitement in the real worldanon2015/07/21 06:22 AM
                                              Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 07:36 AM
                                                Example of memory ordering causing excitement in the real worldanon2015/07/21 08:25 AM
                                              Example of memory ordering causing excitement in the real worldanon2015/07/21 08:17 AM
                                                Example of memory ordering causing excitement in the real worldanon2015/07/21 09:50 AM
                                                Example of memory ordering causing excitement in the real worldGabriele Svelto2015/07/21 12:46 PM
                                                Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/22 12:35 AM
                                                  Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/22 01:37 AM
                                                  Example of memory ordering causing excitement in the real worldanon2015/07/23 03:17 AM
                                                    Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 04:59 AM
                                                      Example of memory ordering causing excitement in the real worldMichael S2015/07/23 07:31 AM
                                                        Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 03:38 PM
                                                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 11:44 PM
                                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/24 01:08 AM
                                    Example of memory ordering causing excitement in the real worlddmcq2015/07/13 07:27 AM
                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/16 05:11 AM
                      Example of memory ordering causing excitement in the real worldAnonymousse2015/07/16 05:07 AM
          Benefit of dynamic hammock predicationsomeotherdude2015/07/06 05:12 PM
            Benefit of dynamic hammock predicationMichael S2015/07/07 01:19 AM
          Benefit of dynamic hammock predicationPatrick Chase2015/07/06 10:20 PM
            SIMD predication is also helpful, of course (NT)Paul A. Clayton2015/07/07 07:45 AM
        Power8 branch conversionGabriele Svelto2015/07/07 02:50 AM
          isel works for storesPaul A. Clayton2015/07/07 07:43 AM
            isel works for storesPatrick Chase2015/07/07 11:08 AM
            isel DOES NOT work for storessomeotherdude2015/07/07 04:01 PM
              isel DOES NOT work for storesWilco2015/07/07 04:40 PM
              Yes it doesPatrick Chase2015/07/07 11:14 PM
                Yes it doesvvid2015/07/08 09:51 AM
            You don't need isel eitherGabriele Svelto2015/07/08 02:01 AM
              You don't need isel eitherPaul A. Clayton2015/07/08 04:46 AM
                You don't need isel eitherGabriele Svelto2015/07/08 05:22 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell tangerine? 🍊