Example of memory ordering causing excitement in the real world

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), July 9, 2015 11:14 am
Room: Moderated Discussions
Maynard Handley (name99.delete@this.name99.org) on July 9, 2015 10:12 am wrote:
>
> In a talk by Leslie Lamport at Stanford a few months ago, he claimed that formal verification of
> an implementation of the POWER memory model had caught a bug that testing would not have caught.
> You can read this as "the POWER memory model sucks" or you can read this as "this
> sort of thing should be handled by formal verification rather than testing"...

I think formal verification is absolutely the way to go when it comes to memory ordering. Because it is too damn hard any other way.

However, I happen to believe in formal verification as a solution for things like cache protocol definitions (where it is already used) and hardware (where I think it's used at some levels). Make the rules clear, understandable and unambiguous (and very much not surprising), and show that the hardware actually follows the rules.

But formal verification is not necessarily nearly as useful for software.

I used to think that it was completely unrealistic for software, but it turns out that it's at least somewhat possible to maybe not really "formally" verify these things, but at least automate a lot of memory access verification, and there are groups doing some very impressive things.

The software verification is often with adding annotations or simplifying the problem space other ways, though - so the "verification" is not necessarily verifying the actual totality code, but a simplified representation or sub-case. In most cases the verification is run over traces of code (running on different CPU's) and verifying that those traces follow the rules. So it may not be formally verifying the whole thing, but at least it's verifying something, preferably against the actual hardware rules that have also been verified to be honored by the hardware.

But that kind of software verification will not be run on most software. There are groups using Linux for their work (and presumably other systems too, I just wouldn't know about it), and there are groups (often the same group) looking at things like core system libraries. So hopefully there will be coverage of the really core system code, and hopefully at least the core locking primitives have been verified to work, and some really basic libraries are using them correctly.

But verifying software (even with the caveats mentioned above) formally in general? Yeah, not going to happen. People will forget to take locks, and people will create their own ad-hoc synchronization models, and they will get them wrong. And they will "work". Most of the time.

And the subtler and the less intuitive the hardware rules are, the more likely it will be that they will work fine on one piece of hardware (inevitably the one that the developer has, because that's what he tested his fancy new idea on), but then result in very odd and occasional problems on other implementations.

Tightening down the hardware rules really does help enormously with problems like that. The code may be buggy as hell, but at least what you tested on one machine tends to be reasonably close to what you run three generations later and your binary still works, rather than mysteriously crashing in ways you can't reproduce.

And btw, this is not at all a new issue. The whole "one CPU does something subtly different from another CPU because it wasn't well-defined in the architecture" is age-old, and it has been a problem before.

Think "undefined flags after operations" kind of things (x86 had several cases of that), or unused high bits in pointer uses (m68k), or incomplete instruction decoding (6502) or silent odd behavior with unaligned addresses (ARM) or any number of those under-specified hardware issues that people ended up using even though they often weren't documented (and then people had to document them).

HW designers have largely learnt not to do that any more. Weak memory ordering isn't really all that different. It's simply worth making the definition stricter (and using extra hardware to enforce the stricter rules) to avoid problems down the line in the next microarchitecture when you changed something fundamentally internally but don't want that change to break existing applications.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cyclone Macro-op fusionMaynard Handley2015/07/05 05:01 PM
  Cyclone Macro-op fusionGabriele Svelto2015/07/06 01:11 AM
    Cyclone Macro-op fusionMaynard Handley2015/07/06 09:25 AM
      Cyclone Macro-op fusionWilco2015/07/06 10:17 AM
        Cyclone Macro-op fusionMaynard Handley2015/07/06 11:07 AM
          Cyclone Macro-op fusionMaynard Handley2015/07/06 11:55 AM
            Cyclone Macro-op fusionGabriele Svelto2015/07/07 01:49 AM
      Power8 branch conversionSHK2015/07/06 11:41 AM
        Power8 branch conversionMaynard Handley2015/07/06 12:01 PM
          Power8 branch conversionMaynard Handley2015/07/06 12:09 PM
        Benefit of dynamic hammock predicationPaul A. Clayton2015/07/06 02:02 PM
          Benefit of dynamic hammock predicationMaynard Handley2015/07/06 02:34 PM
            Benefit of dynamic hammock predicationEduardoS2015/07/06 06:56 PM
            Benefit of dynamic hammock predicationPatrick Chase2015/07/06 09:30 PM
          Benefit of dynamic hammock predicationLinus Torvalds2015/07/06 02:59 PM
            Benefit of dynamic hammock predicationWilco2015/07/06 03:35 PM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/06 04:25 PM
                Benefit of dynamic hammock predicationanon2015/07/06 08:02 PM
            Benefit of dynamic hammock predicationPatrick Chase2015/07/06 09:31 PM
              Benefit of dynamic hammock predicationanon2015/07/06 10:02 PM
              Benefit of dynamic hammock predicationMichael S2015/07/07 12:07 AM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/07 08:23 AM
                Benefit of dynamic hammock predicationPatrick Chase2015/07/07 09:49 AM
                  Benefit of dynamic hammock predicationSHK2015/07/07 10:34 AM
                  Benefit of dynamic hammock predicationMaynard Handley2015/07/07 11:00 AM
                    Benefit of dynamic hammock predicationExophase2015/07/07 12:26 PM
                      Benefit of dynamic hammock predicationWilco2015/07/07 03:54 PM
                      Benefit of dynamic hammock predicationMaynard Handley2015/07/07 04:25 PM
                    Benefit of dynamic hammock predicationPatrick Chase2015/07/07 01:32 PM
                    Benefit of dynamic hammock predicationMichael S2015/07/08 12:57 AM
                  Benefit of dynamic hammock predicationSylvain Collange2015/07/08 05:30 AM
                    Benefit of dynamic hammock predicationMaynard Handley2015/07/08 08:46 AM
                      Benefit of dynamic hammock predicationSylvain Collange2015/07/08 09:32 AM
                        Benefit of dynamic hammock predicationMaynard Handley2015/07/08 09:57 AM
                        Sadly, skewed associativity has not been broadly adoptedPaul A. Clayton2015/07/08 01:00 PM
                        Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/08 07:23 PM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)David Kanter2015/07/09 06:20 AM
                            Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/09 09:04 AM
                              Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Tim McCaffrey2015/07/09 09:41 AM
                                Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 12:08 PM
                                  Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Sylvain Collange2015/07/10 01:03 AM
                              Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)David Kanter2015/07/09 03:15 PM
                                Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Maynard Handley2015/07/09 04:21 PM
                            Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 11:44 AM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)Patrick Chase2015/07/09 08:43 AM
                          Prediction quality of A8 and Intel (was Benefit of dynamic hammock predication)someone2015/07/10 04:43 AM
            Benefit of dynamic hammock predicationKonrad Schwarz2015/07/07 06:54 AM
            Benefit of dynamic hammock predicationdmcq2015/07/07 01:38 PM
              Benefit of dynamic hammock predicationLinus Torvalds2015/07/08 07:23 AM
                Benefit of dynamic hammock predicationdmcq2015/07/08 09:11 AM
                  Benefit of dynamic hammock predicationLinus Torvalds2015/07/08 12:38 PM
                    Example of memory ordering causing excitement in the real worldMark Roulo2015/07/08 03:54 PM
                      Example of memory ordering causing excitement in the real worlddmcq2015/07/09 03:37 AM
                        Example of memory ordering causing excitement in the real worldMark Roulo2015/07/09 07:24 AM
                          Example of memory ordering causing excitement in the real worlddmcq2015/07/09 09:09 AM
                          Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/09 09:12 AM
                            Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/09 11:14 AM
                              Example of memory ordering causing excitement in the real worlddmcq2015/07/09 03:24 PM
                                Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/10 12:07 PM
                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/10 02:49 PM
                                    Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/12 12:02 PM
                                      Example of memory ordering causing excitement in the real worlddmcq2015/07/13 04:03 AM
                                        Example of memory ordering causing excitement in the real worldMichael S2015/07/13 04:19 AM
                                        Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/14 12:21 PM
                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/14 01:54 PM
                                            Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/15 12:13 PM
                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/15 02:43 PM
                                                Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/16 11:41 AM
                                                  Example of memory ordering causing excitement in the real worldWilco2015/07/17 12:57 PM
                                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/17 03:29 PM
                                                      Example of memory ordering causing excitement in the real worlddmcq2015/07/18 03:07 AM
                                          Example of memory ordering causing excitement in the real worldJouni Osmala2015/07/16 02:52 AM
                                            Example of memory ordering causing excitement in the real worldSimon Farnsworth2015/07/16 12:05 PM
                                              Example of memory ordering causing excitement in the real worldJouni Osmala2015/07/17 12:18 AM
                                                ISA support for bound checkingMichael S2015/07/17 03:51 AM
                                                  ISA support for bound checkingEtienne2015/07/17 05:39 AM
                                                  MPX on SkylakeMark Roulo2015/07/17 07:19 AM
                                                    MPX on SkylakeMichael S2015/07/17 07:44 AM
                                                      MPX on SkylakeMark Roulo2015/07/17 08:03 AM
                                  Example of memory ordering causing excitement in the real worldanon2015/07/12 02:42 AM
                                    Example of memory ordering causing excitement in the real worlddmcq2015/07/12 09:27 AM
                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 10:24 AM
                                      Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 10:53 AM
                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 12:07 PM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/12 06:52 PM
                                            Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/13 10:05 AM
                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/13 11:20 AM
                                                Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/13 12:46 PM
                                                  Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/13 01:10 PM
                                                    Example of memory ordering causing excitement in the real worldanon2015/07/13 07:51 PM
                                                    Simple cores justifying weak ordering?Paul A. Clayton2015/07/14 08:01 AM
                                                      David's Paul A. Clayton2015/07/14 08:37 AM
                                                      David's 2015-07-12 10:34 post confirmed some of that (NT)Paul A. Clayton2015/07/14 08:39 AM
                                                    Example of memory ordering causing excitement in the real worldEtienne2015/07/17 12:15 AM
                                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/13 01:14 PM
                                                  Reality is even worseTerry Gray2015/07/13 05:05 PM
                                                    Reality is even worseKonrad Schwarz2015/07/20 11:13 PM
                                                  A simple CAS/MCAS semantics2015/07/15 11:54 AM
                                                    A simple CAS/MCAS semanticsLinus Torvalds2015/07/15 12:55 PM
                                                    A simple CAS/MCAS semanticsEduardoS2015/07/15 01:31 PM
                                                Example of memory ordering causing excitement in the real worldsylt2015/07/13 01:19 PM
                                                  Example of memory ordering causing excitement in the real worlddmcq2015/07/13 02:24 PM
                                                    Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 03:48 PM
                                                      Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/13 06:18 PM
                                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 06:57 PM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/13 08:01 PM
                                                    What high level languages do you have in mind?Mark Roulo2015/07/13 04:00 PM
                                                      What high level languages do you have in mind?dmcq2015/07/14 05:54 AM
                                                  Example of memory ordering causing excitement in the real worldWilco2015/07/13 03:36 PM
                                                    Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 03:56 PM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/13 05:43 PM
                                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/13 07:11 PM
                                                          Example of memory ordering causing excitement in the real worldanon2015/07/13 08:07 PM
                                      Example of memory ordering causing excitement in the real worldMaynard Handley2015/07/12 11:32 AM
                                        Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 12:35 PM
                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/12 12:38 PM
                                          Example of memory ordering causing excitement in the real worldEduardoS2015/07/12 05:47 PM
                                            Example of memory ordering causing excitement in the real worldNoSpammer2015/07/13 11:34 AM
                                              Example of memory ordering causing excitement in the real worldWilco2015/07/13 03:50 PM
                                                Example of memory ordering causing excitement in the real worldNoSpammer2015/07/14 01:02 PM
                                                  Example of memory ordering causing excitement in the real worldanon2015/07/14 08:37 PM
                                                    Example of memory ordering causing excitement in the real worldMichael S2015/07/15 12:19 AM
                                                      Example of memory ordering causing excitement in the real worldanon2015/07/15 01:26 AM
                                                        Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 06:05 AM
                                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/15 07:04 AM
                                                            Example of memory ordering causing excitement in the real worldMichael S2015/07/15 07:29 AM
                                                              Example of memory ordering causing excitement in the real worlddmcq2015/07/15 09:09 AM
                                                              Can string ops use microarchitectural state?Paul A. Clayton2015/07/16 08:33 AM
                                                                Can string ops use microarchitectural state?rwessel2015/07/16 12:00 PM
                                                                  Can string ops use microarchitectural state?Paul A. Clayton2015/07/16 05:16 PM
                                                                    Can string ops use microarchitectural state?rwessel2015/07/16 08:34 PM
                                                                      Can string ops use microarchitectural state?Michael S2015/07/17 04:39 AM
                                                                        Can string ops use microarchitectural state?rwessel2015/07/17 08:02 AM
                                                                Consider interrupt followed by task preemptionMichael S2015/07/16 03:31 PM
                                                                  Consider interrupt followed by task preemptionPaul A. Clayton2015/07/16 05:23 PM
                                                            Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 11:50 AM
                                                              Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 01:27 PM
                                                                Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 02:14 PM
                                                                  Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 04:42 PM
                                                      Example of memory ordering causing excitement in the real worldNoSpammer2015/07/15 01:44 AM
                                                    Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 01:17 PM
                                                      Example of memory ordering causing excitement in the real worldWilco2015/07/15 01:43 PM
                                                        Example of memory ordering causing excitement in the real worldanon2015/07/15 04:45 PM
                                                        Example of memory ordering causing excitement in the real worldLinus Torvalds2015/07/15 05:35 PM
                                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/16 01:46 AM
                                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/16 01:49 AM
                                        Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 06:15 AM
                                      Example of memory ordering causing excitement in the real worldanon2015/07/12 06:42 PM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/12 08:48 PM
                                        Example of memory ordering causing excitement in the real worldDavid Kanter2015/07/12 09:34 PM
                                          Example of memory ordering causing excitement in the real worldMichael S2015/07/13 12:51 AM
                                            Example of memory ordering causing excitement in the real worldDavid Kanter2015/07/13 07:31 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/13 05:05 AM
                                            Example of memory ordering causing excitement in the real worldGabriele Svelto2015/07/17 04:40 AM
                                          Example of memory ordering causing excitement in the real worlddmcq2015/07/13 07:09 AM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/13 06:46 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/14 03:12 AM
                                        Example of memory ordering causing excitement in the real worlddmcq2015/07/15 02:51 AM
                                          Example of memory ordering causing excitement in the real worldanon2015/07/15 03:38 AM
                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/15 05:21 AM
                                      Store queues in POWER/PowerPC processorsGabriele Svelto2015/07/16 12:55 AM
                                        Not sure about that conclusionDavid Kanter2015/07/16 06:34 AM
                                          The POWER4 had the same structureGabriele Svelto2015/07/17 04:20 AM
                                        Committing stores in program orderPaul A. Clayton2015/07/16 12:35 PM
                                          Committing stores in program orderMaynard Handley2015/07/17 11:52 AM
                                      Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/20 03:44 AM
                                        Example of memory ordering causing excitement in the real worldanon2015/07/20 06:29 AM
                                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/20 11:08 PM
                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/21 12:27 AM
                                            Example of memory ordering causing excitement in the real worldMichael S2015/07/21 02:11 AM
                                              Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 04:24 AM
                                            Example of memory ordering causing excitement in the real worldanon2015/07/21 05:22 AM
                                              Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 06:36 AM
                                                Example of memory ordering causing excitement in the real worldanon2015/07/21 07:25 AM
                                              Example of memory ordering causing excitement in the real worldanon2015/07/21 07:17 AM
                                                Example of memory ordering causing excitement in the real worldanon2015/07/21 08:50 AM
                                                Example of memory ordering causing excitement in the real worldGabriele Svelto2015/07/21 11:46 AM
                                                Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/21 11:35 PM
                                                  Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/22 12:37 AM
                                                  Example of memory ordering causing excitement in the real worldanon2015/07/23 02:17 AM
                                                    Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 03:59 AM
                                                      Example of memory ordering causing excitement in the real worldMichael S2015/07/23 06:31 AM
                                                        Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 02:38 PM
                                                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/23 10:44 PM
                                                            Example of memory ordering causing excitement in the real worlddmcq2015/07/24 12:08 AM
                                    Example of memory ordering causing excitement in the real worlddmcq2015/07/13 06:27 AM
                          Example of memory ordering causing excitement in the real worldKonrad Schwarz2015/07/16 04:11 AM
                      Example of memory ordering causing excitement in the real worldAnonymousse2015/07/16 04:07 AM
          Benefit of dynamic hammock predicationsomeotherdude2015/07/06 04:12 PM
            Benefit of dynamic hammock predicationMichael S2015/07/07 12:19 AM
          Benefit of dynamic hammock predicationPatrick Chase2015/07/06 09:20 PM
            SIMD predication is also helpful, of course (NT)Paul A. Clayton2015/07/07 06:45 AM
        Power8 branch conversionGabriele Svelto2015/07/07 01:50 AM
          isel works for storesPaul A. Clayton2015/07/07 06:43 AM
            isel works for storesPatrick Chase2015/07/07 10:08 AM
            isel DOES NOT work for storessomeotherdude2015/07/07 03:01 PM
              isel DOES NOT work for storesWilco2015/07/07 03:40 PM
              Yes it doesPatrick Chase2015/07/07 10:14 PM
                Yes it doesvvid2015/07/08 08:51 AM
            You don't need isel eitherGabriele Svelto2015/07/08 01:01 AM
              You don't need isel eitherPaul A. Clayton2015/07/08 03:46 AM
                You don't need isel eitherGabriele Svelto2015/07/08 04:22 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?