ARMv8 getting atomic operations

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), December 4, 2014 12:05 pm
Room: Moderated Discussions
anon (anon.delete@this.anon.com) on December 3, 2014 5:08 pm wrote:
>
> We've talked about this before here, but LL/SC can guarantee progress (when it is limited
> like POWER does), and the LL of course always carries a load-for-store signal.

Both of these are "true", but not what I'm complaining about.

First off, "guaranteed forward progress" in LL/SC tends to be a global thing: you're guaranteeing not to livelock. That is not interesting from a performance standpoint, it's just interesting from a "minimal requirements" standpoint.

The thing is, the LL/SC model (and the load/cmpxchg model) does not guarantee that each thread makes forward progress, much less that each cache miss makes any progress.

Seriously, it's a real issue. The cmpxchg fails. Not just occasionally. Under real load (admittedly very high contention), it fails quite often. And each failure is basically an extra and unnecessary ping-pong of a cacheline, where some other CPU ended up winning a race, and causing the cache access on the losing CPU to be pure and utter useless work.

On x86 (which, again, is the only architecture you can actually compare these approaches), the numbers seem to be that if you update counters, the atomic RMW "add" model gets about twice the progress over a "load+add+cmpxchg" model. Twice.

And yes, that obviously ends up depending on cache coherency details etc, and how "sticky" a cacheline is to a CPU that got it. But that's actually a real potential issue too: being too sticky tends to improve throughput, but can cause some seriously excessive unfairness issues, where a node or a core that got exclusive access to the cacheline and keeps writing to it can get very unfair advantages wrt other cores.

And yes, we've very much seen that too especially in NUMA environments.

So LL/SC either fails a lot and causes ping-pong traffic while only making very slow progress (some progress, yes), or can try to avoid the failure case by making the cachelines very sticky and then become very unfair and amenable to imbalance.

A RMW model doesn't tend to have the same kind of issues. Once you got the cacheline, you will update it. There is no failure case and unnecessary cache transaction.

And yes, you can in theory make LL/SC or load/cmpxchg work like a RMW by noticing the pattern and basically turning the bad LL/SC model into an RMW model by generating macro-instructions. And I actually think it's not a bad idea. But even if you do that, you basically have to first admit the superiority of the RMW model.

As to the LL always implying write intent, I agree that it tends to make more sense. I'm not actually convinced everybody always does that. In the x86 world, where the pseudo-equivalent sequence is load/cmpxchg, we definitely have hit that issue.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
ARMv8 getting atomic operationsdmcq12/02/14 05:32 PM
  ARMv8 getting atomic operationsMaynard Handley12/02/14 07:33 PM
    ARMv8 getting atomic operationsDoug S12/02/14 10:30 PM
      ARMv8 getting atomic operationsdmcq12/03/14 03:16 AM
      ARMv8 getting atomic operationsMaynard Handley12/03/14 09:20 AM
      ARMv8 getting atomic operationsBrett12/03/14 04:46 PM
    ARMv8 getting atomic operationsAndreas12/03/14 06:51 AM
      ARMv8 getting atomic operationsLinus Torvalds12/03/14 11:15 AM
        ARMv8 getting atomic operationsanon12/03/14 05:08 PM
          Guaranteed transactionsPaul A. Clayton12/03/14 08:04 PM
            Guaranteed transactionsanon12/03/14 08:38 PM
              Avoiding ping pongPaul A. Clayton12/04/14 09:11 AM
                Avoiding ping ponganon12/04/14 10:15 AM
                  OoO window is limitedPaul A. Clayton12/04/14 01:06 PM
                Avoiding ping pongAaron Spink12/04/14 12:01 PM
                  Avoiding ping pongKonrad Schwarz12/04/14 01:10 PM
                    Avoiding ping pongAaron Spink12/04/14 02:31 PM
                    Avoiding ping pongGabriele Svelto12/04/14 02:49 PM
                      Avoiding ping pongKonrad Schwarz12/04/14 11:08 PM
                        Avoiding ping pongGabriele Svelto12/05/14 12:04 AM
                          Avoiding ping pongEric Bron nli12/05/14 02:28 AM
                            Avoiding ping pongKonrad Schwarz12/05/14 03:37 AM
                              Avoiding ping pongEric Bron nli12/05/14 04:23 AM
                                Avoiding ping pongKlimax12/05/14 05:47 AM
                                  Avoiding ping pongEric Bron12/05/14 06:24 AM
                              Avoiding ping pongGabriele Svelto12/05/14 10:38 AM
                                Avoiding ping pongKonrad Schwarz12/07/14 02:28 PM
                                  Avoiding ping pongGabriele Svelto12/08/14 07:10 PM
                                    Avoiding ping pongKonrad Schwarz12/09/14 05:12 AM
                                      Avoiding ping pongGabriele Svelto12/09/14 07:31 AM
                                        Avoiding ping ponganon12/09/14 11:24 PM
                            Avoiding ping pongGabriele Svelto12/05/14 10:17 AM
                              Avoiding ping pongEric Bron12/05/14 10:32 AM
                                Avoiding ping pongGabriele Svelto12/05/14 12:45 PM
                                  Avoiding ping pongEric Bron12/06/14 02:20 AM
                                    Avoiding ping pongnksingh12/06/14 03:42 AM
                                      Avoiding ping pongEric Bron12/06/14 04:04 AM
                                        Avoiding ping pongGiGNiC12/06/14 06:27 AM
                                          Avoiding ping pongEric Bron nli12/06/14 06:44 AM
                                          Avoiding ping pongEric Bron12/06/14 07:07 AM
                                            Avoiding ping pongnksingh12/07/14 04:06 PM
                                              Avoiding ping pongEric Bron12/08/14 04:17 AM
                                                Avoiding ping pongGiGNiC12/08/14 11:53 AM
                                                Avoiding ping pongnksingh12/08/14 05:53 PM
                                                  Avoiding ping pongEric Bron12/09/14 01:33 AM
                                    Avoiding ping pongdmsc12/06/14 04:12 AM
                                      Avoiding ping pongEric Bron12/06/14 04:25 AM
                                        Avoiding ping pongKlimax12/06/14 05:49 AM
                                          Avoiding ping pongrwessel12/07/14 02:34 AM
                                        Avoiding ping pongdmsc12/06/14 07:39 AM
                                        Avoiding ping pongKonrad Schwarz12/07/14 02:37 PM
                                          Avoiding ping pongMichael S12/07/14 04:37 PM
                                            Avoiding ping pongKonrad Schwarz12/08/14 04:35 AM
                          Avoiding ping pongKonrad Schwarz12/05/14 03:30 AM
                        Avoiding ping pongLinus Torvalds12/05/14 12:58 PM
                          Avoiding ping pongEric Bron12/06/14 02:42 AM
                            Avoiding ping pongnksingh12/06/14 03:51 AM
                              Avoiding ping pongEric Bron12/06/14 04:08 AM
                            Avoiding ping pongLinus Torvalds12/06/14 01:25 PM
                              Avoiding ping pongnksingh12/07/14 03:26 PM
                                Avoiding ping pongEric Bron12/08/14 04:35 AM
                                  Avoiding ping pongBrett12/08/14 10:00 AM
                                    Avoiding ping pongEric Bron12/08/14 10:48 AM
                                    Avoiding ping pongrwessel12/08/14 12:52 PM
                                      Avoiding ping pongBrett12/08/14 01:58 PM
                                      Avoiding ping pongDoug S12/08/14 02:04 PM
                              Avoiding ping pongJouni Osmala12/08/14 02:45 AM
                                Avoiding ping ponganon12/08/14 05:44 AM
                                  Avoiding ping pongJouni Osmala12/08/14 01:10 PM
                                    Avoiding ping pongLinus Torvalds12/08/14 01:34 PM
                                      Avoiding ping pongJouni Osmala12/08/14 03:47 PM
                                        Avoiding ping pongLinus Torvalds12/08/14 08:08 PM
                                          Avoiding ping pongGabriele Svelto12/09/14 07:48 AM
                                            Avoiding ping pongMaynard Handley12/09/14 11:41 AM
                                              Avoiding ping pongPatrick Chase12/09/14 01:06 PM
                                              Avoiding ping pongGabriele Svelto12/09/14 01:52 PM
                                                Avoiding ping pongPatrick Chase12/09/14 02:08 PM
                                            Why read RWT or Reddit when you can get journalists to do it for you?Rob Thorpe01/02/15 08:20 AM
                                              Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/02/15 11:21 AM
                                                Why read RWT or Reddit when you can get journalists to do it for you?EduardoS01/02/15 11:37 AM
                                                  Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 12:00 PM
                                                Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron nli01/02/15 02:28 PM
                                                  Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 12:02 PM
                                                    Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/03/15 12:36 PM
                                                      Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 01:11 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/03/15 01:30 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:57 PM
                                                            KNL cacheDavid Kanter01/03/15 07:36 PM
                                                              KNL cacheEric Bron01/04/15 03:34 AM
                                                                KNL cacheMichael S01/04/15 04:11 AM
                                                                  KNL cacheEric Bron01/04/15 04:57 AM
                                                                    KNL cacheMichael S01/04/15 05:21 AM
                                                                      KNL cacheEric Bron01/04/15 05:58 AM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 05:47 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/07/15 08:27 AM
                                                              Manycores vs multicoresjuanrga01/10/15 04:10 PM
                                                                Manycores vs multicoresAaron Spink01/10/15 05:32 PM
                                                                  Manycores vs multicoresjuanrga01/10/15 06:32 PM
                                                                    Manycores vs multicoresExophase01/10/15 06:49 PM
                                                                      Manycores vs multicoresjuanrga01/10/15 08:21 PM
                                                                        Manycores vs multicoresExophase01/10/15 08:51 PM
                                                                        Manycores vs multicoresAaron Spink01/10/15 09:03 PM
                                                                    Manycores vs multicoresAaron Spink01/10/15 07:21 PM
                                                                      Manycores vs multicoresjuanrga01/10/15 08:25 PM
                                                                        Manycores vs multicoresAaron Spink01/10/15 09:11 PM
                                                                          Manycores vs multicoresJouni Osmala01/11/15 04:50 AM
                                                                            Manycores vs multicoresjuanrga01/11/15 08:58 AM
                                                                            Manycores vs multicorescoppice01/12/15 10:01 PM
                                                                              Manycores vs multicoresJouni Osmala01/13/15 04:38 AM
                                                                        Manycores vs multicoresanon01/11/15 03:19 AM
                                                                      Manycores vs multicoresMichael S01/11/15 05:44 AM
                                                                        Manycores vs multicoresAaron Spink01/11/15 05:55 PM
                                                                          Manycores vs multicoresMichael S01/12/15 04:41 AM
                                                                            Manycores vs multicoresEric Bron01/12/15 06:29 AM
                                                                              Manycores vs multicoresEric Bron01/12/15 06:30 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:54 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 05:48 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/07/15 07:41 AM
                                                              Manycores vs multicoresjuanrga01/10/15 04:14 PM
                                                    Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:42 PM
                                                      Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 06:03 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/07/15 07:45 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Linus Torvalds01/08/15 03:09 PM
                                                          Pink unicorns for salejuanrga01/10/15 05:09 PM
                                                        Intentionally picking a competitors slow part is cheating ...Mark Roulo01/08/15 06:37 PM
                                                          Intentionally picking a competitors slow part is cheating ...coppice01/08/15 11:38 PM
                                                            Intentionally picking a competitors slow part is cheating ...Mark Roulo01/09/15 09:13 AM
                                                              Intentionally picking a competitors slow part is cheating ...Anon01/10/15 02:00 AM
                                                              Intentionally picking a competitors slow part is cheating ...David Hess01/11/15 01:03 PM
                                                            Intentionally picking a competitors slow part is cheating ...someone01/09/15 10:31 AM
                                                              Intentionally picking a competitors slow part is cheating ...coppice01/12/15 09:45 PM
                                                                Intentionally picking a competitors slow part is cheating ...coppice01/12/15 09:47 PM
                                                                  Intentionally picking a competitors slow part is cheating ...Michael S01/13/15 07:53 AM
                                                                    Intentionally picking a competitors slow part is cheating ...coppice01/13/15 09:44 AM
                                                                      Intentionally picking a competitors slow part is cheating ...Michael S01/13/15 10:01 AM
                                                                        Intentionally picking a competitors slow part is cheating ...coppice01/13/15 08:35 PM
                                                                      Core sizesjuanrga01/13/15 12:28 PM
                                                          NVIDIA'S FIRST CPU IS A WINNER (Linley Gwennap)juanrga01/10/15 04:34 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/08/15 07:02 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?coppice01/08/15 10:18 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/09/15 11:54 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Mark Roulo01/09/15 12:59 PM
                                                              Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/09/15 03:20 PM
                                                                Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/09/15 03:30 PM
                                                            Alternatives to OOOE (again)juanrga01/10/15 04:50 PM
                                                              Alternatives to OOOE (again)David Kanter01/11/15 12:10 AM
                                                                Alternatives to OOOE (again)juanrga01/11/15 08:30 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Gabriele Svelto01/11/15 12:53 AM
                                              Why read RWT or Reddit when you can get journalists to do it for you?Fake Linus Torvalds01/03/15 12:14 PM
                                                Why read RWT or Reddit when you can get journalists to do it for you?Rob Thorpe01/03/15 08:25 PM
                                          Avoiding ping pongMaynard Handley12/09/14 11:33 AM
                                            Avoiding ping pongPatrick Chase12/09/14 01:54 PM
                                              Avoiding ping pongMaynard Handley12/09/14 06:56 PM
                                      Avoiding ping pongSalvatore De Dominicis12/09/14 08:51 AM
                                        Avoiding ping pongPatrick Chase12/09/14 02:00 PM
                                      Avoiding ping pongook12/11/14 03:31 AM
                                      Avoiding ping pongArt Scott12/19/14 10:19 PM
                                        Avoiding ping pongEric Bron nli12/20/14 04:05 AM
                                      What about specialization?Troll?01/02/15 07:55 AM
                                        What about specialization?Ungo01/04/15 03:27 PM
                                      Avoiding ping pongfewwef01/05/15 08:16 PM
                                      Avoiding ping pongV.Krishn01/08/15 06:11 AM
                                    Avoiding ping pongGabriele Svelto12/08/14 07:32 PM
                                    Avoiding ping ponganon12/08/14 11:37 PM
                            Avoiding ping pongKonrad Schwarz12/10/14 06:23 AM
                              Avoiding ping pongLinus Torvalds12/10/14 11:56 AM
                          Object reference lockingDavid W12/08/14 11:36 PM
                            Object reference lockingPatrick Chase12/09/14 04:52 PM
                              Object reference lockingDavid W12/11/14 05:18 AM
                    ISA != interface for "most programmers"Paul A. Clayton12/04/14 03:34 PM
                      ISA != interface for "most programmers"rwessel12/04/14 07:50 PM
                  Interesting! (exporting hot lines/cache-aware ISA); "Please sir, I want some more" (NT)Paul A. Clayton12/04/14 02:26 PM
                  Avoiding ping pongMichael S12/06/14 03:48 PM
          ARMv8 getting atomic operationsLinus Torvalds12/04/14 12:05 PM
            LL/SC idiom recognition is not admitting RMW superiorityPaul A. Clayton12/04/14 02:34 PM
            ARMv8 getting atomic operationsanon12/04/14 10:17 PM
    ARMv8 getting atomic operationsPatrick Chase12/03/14 12:09 PM
  limited ordernksingh12/04/14 10:17 PM
    I didn't understand this either. (NT)Konrad Schwarz12/04/14 10:32 PM
    limited orderdmcq12/05/14 02:13 AM
    limited orderbakaneko12/05/14 09:11 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?