ARMv8 getting atomic operations

By: anon (anon.delete@this.anon.com), December 4, 2014 10:17 pm
Room: Moderated Discussions
Linus Torvalds (torvalds.delete@this.linux-foundation.org) on December 4, 2014 12:05 pm wrote:
> anon (anon.delete@this.anon.com) on December 3, 2014 5:08 pm wrote:
> >
> > We've talked about this before here, but LL/SC can guarantee progress (when it is limited
> > like POWER does), and the LL of course always carries a load-for-store signal.
>
> Both of these are "true", but not what I'm complaining about.
>
> First off, "guaranteed forward progress" in LL/SC tends to be a global thing: you're
> guaranteeing not to livelock. That is not interesting from a performance standpoint,
> it's just interesting from a "minimal requirements" standpoint.

Atomic RMW operations do not guarantee individual forward progress either. Some x86 implementations have had near-livelock or at least significant unfairness in them.

For a *constrained* LL/SC operation, ensuring forward progress and fairness is not such a problem in the core. It's in the coherency protocol, and in that case there is not much difference between them (either way the operation will be that the core requests the cacheline exclusive loads from it, performs and operation, and stores the result back, without relinquishing it).

>
> The thing is, the LL/SC model (and the load/cmpxchg model) does not guarantee that each
> thread makes forward progress, much less that each cache miss makes any progress.

A *constrained* LL/SC is much less like an (unconstrained) load and eventual cmpxchg, and much more like just the cmpxchg itself (or the fetch add, or decrement and test, etc).

I agree that completely unconstrained LL/SC are not viable, but that's not how they are used and not how they were intended to be used. They're a RISC-y kind of approach to building the compliment of basic read-modify-write operations.

> Seriously, it's a real issue. The cmpxchg fails. Not just occasionally. Under real load
> (admittedly very high contention), it fails quite often. And each failure is basically an
> extra and unnecessary ping-pong of a cacheline, where some other CPU ended up winning a
> race, and causing the cache access on the losing CPU to be pure and utter useless work.

Yes, and that's a disadvantage of x86. It has to use load + cmpxchg to fill in the gaps of its atomic primitives, and load + cmpxchg is certainly worse than LL/SC for doing that job. Arguably the core could do some fancy detection of the load, but it's significantly harder than for a constrained LL/SC, I would say.

>
> On x86 (which, again, is the only architecture you can actually compare these approaches),
> the numbers seem to be that if you update counters, the atomic RMW "add" model gets
> about twice the progress over a "load+add+cmpxchg" model. Twice.

That's not a valid comparison, as I said.

>
> And yes, that obviously ends up depending on cache coherency details etc, and how "sticky" a cacheline is
> to a CPU that got it. But that's actually a real potential issue too: being too sticky tends to improve
> throughput, but can cause some seriously excessive unfairness issues, where a node or a core that got exclusive
> access to the cacheline and keeps writing to it can get very unfair advantages wrt other cores.
>
> And yes, we've very much seen that too especially in NUMA environments.
>
> So LL/SC either fails a lot and causes ping-pong traffic while only making very slow
> progress (some progress, yes), or can try to avoid the failure case by making the
> cachelines very sticky and then become very unfair and amenable to imbalance.

That's simply not the case on decent implementations (like POWERx). They won't fail the SC a lot.

>
> A RMW model doesn't tend to have the same kind of issues. Once you got the cacheline,
> you will update it. There is no failure case and unnecessary cache transaction.
>
> And yes, you can in theory make LL/SC or load/cmpxchg work like a RMW by noticing the pattern and basically
> turning the bad LL/SC model into an RMW model by generating macro-instructions. And I actually think it's not
> a bad idea. But even if you do that, you basically have to first admit the superiority of the RMW model.

For constrained LL/SC, I believe that is what POWER CPUs do. I should stop repeating this because I still haven't been able to find the source of my claim, but I'm fairly sure I have read it in IBM publications.

I think it would be relatively easy to implement it with an optimistic first pass -- try to hold the cache line exclusive and avoid exceptions for a small period after LL (enough to avoid invalidation in 99.9x% of cases that you have a constrained LL/SC sequence), and then enter a slower path if the SC fails, which would do more work to guarantee it.

> As to the LL always implying write intent, I agree that it tends to make more sense.
> I'm not actually convinced everybody always does that. In the x86 world, where the
> pseudo-equivalent sequence is load/cmpxchg, we definitely have hit that issue.

Well "everybody always does" is not very useful. I'm sure some implementations have really shit atomic RMW instructions too.

IBM does load exclusive with LL of course. They've been improving it too, with the "exclusive access" hint in the LL.

Atomic Update (EH=0)
This hint indicates that the program is using a fetch and
operate (e.g., fetch and add) or some similar algorithm
and that all programs accessing the shared variable are
likely to use a similar operation to access the shared
variable for some time.

Exclusive Access (EH=1)
This hint indicates that the program is attempting to
acquire a lock and if it succeeds, will perform another
store to the lock variable (releasing the lock) before
another program attempts to modify the lock variable."

Not that this is any inherent advantage of LL/SC style (such hints could be put in atomic RMW instructions). Just that IBM is taking this stuff seriously, so they'll certainly have picked up such low hanging fruit as load-for-store.
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
ARMv8 getting atomic operationsdmcq12/02/14 05:32 PM
  ARMv8 getting atomic operationsMaynard Handley12/02/14 07:33 PM
    ARMv8 getting atomic operationsDoug S12/02/14 10:30 PM
      ARMv8 getting atomic operationsdmcq12/03/14 03:16 AM
      ARMv8 getting atomic operationsMaynard Handley12/03/14 09:20 AM
      ARMv8 getting atomic operationsBrett12/03/14 04:46 PM
    ARMv8 getting atomic operationsAndreas12/03/14 06:51 AM
      ARMv8 getting atomic operationsLinus Torvalds12/03/14 11:15 AM
        ARMv8 getting atomic operationsanon12/03/14 05:08 PM
          Guaranteed transactionsPaul A. Clayton12/03/14 08:04 PM
            Guaranteed transactionsanon12/03/14 08:38 PM
              Avoiding ping pongPaul A. Clayton12/04/14 09:11 AM
                Avoiding ping ponganon12/04/14 10:15 AM
                  OoO window is limitedPaul A. Clayton12/04/14 01:06 PM
                Avoiding ping pongAaron Spink12/04/14 12:01 PM
                  Avoiding ping pongKonrad Schwarz12/04/14 01:10 PM
                    Avoiding ping pongAaron Spink12/04/14 02:31 PM
                    Avoiding ping pongGabriele Svelto12/04/14 02:49 PM
                      Avoiding ping pongKonrad Schwarz12/04/14 11:08 PM
                        Avoiding ping pongGabriele Svelto12/05/14 12:04 AM
                          Avoiding ping pongEric Bron nli12/05/14 02:28 AM
                            Avoiding ping pongKonrad Schwarz12/05/14 03:37 AM
                              Avoiding ping pongEric Bron nli12/05/14 04:23 AM
                                Avoiding ping pongKlimax12/05/14 05:47 AM
                                  Avoiding ping pongEric Bron12/05/14 06:24 AM
                              Avoiding ping pongGabriele Svelto12/05/14 10:38 AM
                                Avoiding ping pongKonrad Schwarz12/07/14 02:28 PM
                                  Avoiding ping pongGabriele Svelto12/08/14 07:10 PM
                                    Avoiding ping pongKonrad Schwarz12/09/14 05:12 AM
                                      Avoiding ping pongGabriele Svelto12/09/14 07:31 AM
                                        Avoiding ping ponganon12/09/14 11:24 PM
                            Avoiding ping pongGabriele Svelto12/05/14 10:17 AM
                              Avoiding ping pongEric Bron12/05/14 10:32 AM
                                Avoiding ping pongGabriele Svelto12/05/14 12:45 PM
                                  Avoiding ping pongEric Bron12/06/14 02:20 AM
                                    Avoiding ping pongnksingh12/06/14 03:42 AM
                                      Avoiding ping pongEric Bron12/06/14 04:04 AM
                                        Avoiding ping pongGiGNiC12/06/14 06:27 AM
                                          Avoiding ping pongEric Bron nli12/06/14 06:44 AM
                                          Avoiding ping pongEric Bron12/06/14 07:07 AM
                                            Avoiding ping pongnksingh12/07/14 04:06 PM
                                              Avoiding ping pongEric Bron12/08/14 04:17 AM
                                                Avoiding ping pongGiGNiC12/08/14 11:53 AM
                                                Avoiding ping pongnksingh12/08/14 05:53 PM
                                                  Avoiding ping pongEric Bron12/09/14 01:33 AM
                                    Avoiding ping pongdmsc12/06/14 04:12 AM
                                      Avoiding ping pongEric Bron12/06/14 04:25 AM
                                        Avoiding ping pongKlimax12/06/14 05:49 AM
                                          Avoiding ping pongrwessel12/07/14 02:34 AM
                                        Avoiding ping pongdmsc12/06/14 07:39 AM
                                        Avoiding ping pongKonrad Schwarz12/07/14 02:37 PM
                                          Avoiding ping pongMichael S12/07/14 04:37 PM
                                            Avoiding ping pongKonrad Schwarz12/08/14 04:35 AM
                          Avoiding ping pongKonrad Schwarz12/05/14 03:30 AM
                        Avoiding ping pongLinus Torvalds12/05/14 12:58 PM
                          Avoiding ping pongEric Bron12/06/14 02:42 AM
                            Avoiding ping pongnksingh12/06/14 03:51 AM
                              Avoiding ping pongEric Bron12/06/14 04:08 AM
                            Avoiding ping pongLinus Torvalds12/06/14 01:25 PM
                              Avoiding ping pongnksingh12/07/14 03:26 PM
                                Avoiding ping pongEric Bron12/08/14 04:35 AM
                                  Avoiding ping pongBrett12/08/14 10:00 AM
                                    Avoiding ping pongEric Bron12/08/14 10:48 AM
                                    Avoiding ping pongrwessel12/08/14 12:52 PM
                                      Avoiding ping pongBrett12/08/14 01:58 PM
                                      Avoiding ping pongDoug S12/08/14 02:04 PM
                              Avoiding ping pongJouni Osmala12/08/14 02:45 AM
                                Avoiding ping ponganon12/08/14 05:44 AM
                                  Avoiding ping pongJouni Osmala12/08/14 01:10 PM
                                    Avoiding ping pongLinus Torvalds12/08/14 01:34 PM
                                      Avoiding ping pongJouni Osmala12/08/14 03:47 PM
                                        Avoiding ping pongLinus Torvalds12/08/14 08:08 PM
                                          Avoiding ping pongGabriele Svelto12/09/14 07:48 AM
                                            Avoiding ping pongMaynard Handley12/09/14 11:41 AM
                                              Avoiding ping pongPatrick Chase12/09/14 01:06 PM
                                              Avoiding ping pongGabriele Svelto12/09/14 01:52 PM
                                                Avoiding ping pongPatrick Chase12/09/14 02:08 PM
                                            Why read RWT or Reddit when you can get journalists to do it for you?Rob Thorpe01/02/15 08:20 AM
                                              Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/02/15 11:21 AM
                                                Why read RWT or Reddit when you can get journalists to do it for you?EduardoS01/02/15 11:37 AM
                                                  Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 12:00 PM
                                                Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron nli01/02/15 02:28 PM
                                                  Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 12:02 PM
                                                    Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/03/15 12:36 PM
                                                      Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/03/15 01:11 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/03/15 01:30 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:57 PM
                                                            KNL cacheDavid Kanter01/03/15 07:36 PM
                                                              KNL cacheEric Bron01/04/15 03:34 AM
                                                                KNL cacheMichael S01/04/15 04:11 AM
                                                                  KNL cacheEric Bron01/04/15 04:57 AM
                                                                    KNL cacheMichael S01/04/15 05:21 AM
                                                                      KNL cacheEric Bron01/04/15 05:58 AM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 05:47 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Michael S01/07/15 08:27 AM
                                                              Manycores vs multicoresjuanrga01/10/15 04:10 PM
                                                                Manycores vs multicoresAaron Spink01/10/15 05:32 PM
                                                                  Manycores vs multicoresjuanrga01/10/15 06:32 PM
                                                                    Manycores vs multicoresExophase01/10/15 06:49 PM
                                                                      Manycores vs multicoresjuanrga01/10/15 08:21 PM
                                                                        Manycores vs multicoresExophase01/10/15 08:51 PM
                                                                        Manycores vs multicoresAaron Spink01/10/15 09:03 PM
                                                                    Manycores vs multicoresAaron Spink01/10/15 07:21 PM
                                                                      Manycores vs multicoresjuanrga01/10/15 08:25 PM
                                                                        Manycores vs multicoresAaron Spink01/10/15 09:11 PM
                                                                          Manycores vs multicoresJouni Osmala01/11/15 04:50 AM
                                                                            Manycores vs multicoresjuanrga01/11/15 08:58 AM
                                                                            Manycores vs multicorescoppice01/12/15 10:01 PM
                                                                              Manycores vs multicoresJouni Osmala01/13/15 04:38 AM
                                                                        Manycores vs multicoresanon01/11/15 03:19 AM
                                                                      Manycores vs multicoresMichael S01/11/15 05:44 AM
                                                                        Manycores vs multicoresAaron Spink01/11/15 05:55 PM
                                                                          Manycores vs multicoresMichael S01/12/15 04:41 AM
                                                                            Manycores vs multicoresEric Bron01/12/15 06:29 AM
                                                                              Manycores vs multicoresEric Bron01/12/15 06:30 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:54 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 05:48 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/07/15 07:41 AM
                                                              Manycores vs multicoresjuanrga01/10/15 04:14 PM
                                                    Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/03/15 02:42 PM
                                                      Why read RWT or Reddit when you can get journalists to do it for you?juanrga01/07/15 06:03 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/07/15 07:45 AM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Linus Torvalds01/08/15 03:09 PM
                                                          Pink unicorns for salejuanrga01/10/15 05:09 PM
                                                        Intentionally picking a competitors slow part is cheating ...Mark Roulo01/08/15 06:37 PM
                                                          Intentionally picking a competitors slow part is cheating ...coppice01/08/15 11:38 PM
                                                            Intentionally picking a competitors slow part is cheating ...Mark Roulo01/09/15 09:13 AM
                                                              Intentionally picking a competitors slow part is cheating ...Anon01/10/15 02:00 AM
                                                              Intentionally picking a competitors slow part is cheating ...David Hess01/11/15 01:03 PM
                                                            Intentionally picking a competitors slow part is cheating ...someone01/09/15 10:31 AM
                                                              Intentionally picking a competitors slow part is cheating ...coppice01/12/15 09:45 PM
                                                                Intentionally picking a competitors slow part is cheating ...coppice01/12/15 09:47 PM
                                                                  Intentionally picking a competitors slow part is cheating ...Michael S01/13/15 07:53 AM
                                                                    Intentionally picking a competitors slow part is cheating ...coppice01/13/15 09:44 AM
                                                                      Intentionally picking a competitors slow part is cheating ...Michael S01/13/15 10:01 AM
                                                                        Intentionally picking a competitors slow part is cheating ...coppice01/13/15 08:35 PM
                                                                      Core sizesjuanrga01/13/15 12:28 PM
                                                          NVIDIA'S FIRST CPU IS A WINNER (Linley Gwennap)juanrga01/10/15 04:34 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/08/15 07:02 PM
                                                        Why read RWT or Reddit when you can get journalists to do it for you?coppice01/08/15 10:18 PM
                                                          Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/09/15 11:54 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Mark Roulo01/09/15 12:59 PM
                                                              Why read RWT or Reddit when you can get journalists to do it for you?Patrick Chase01/09/15 03:20 PM
                                                                Why read RWT or Reddit when you can get journalists to do it for you?Eric Bron01/09/15 03:30 PM
                                                            Alternatives to OOOE (again)juanrga01/10/15 04:50 PM
                                                              Alternatives to OOOE (again)David Kanter01/11/15 12:10 AM
                                                                Alternatives to OOOE (again)juanrga01/11/15 08:30 AM
                                                            Why read RWT or Reddit when you can get journalists to do it for you?Gabriele Svelto01/11/15 12:53 AM
                                              Why read RWT or Reddit when you can get journalists to do it for you?Fake Linus Torvalds01/03/15 12:14 PM
                                                Why read RWT or Reddit when you can get journalists to do it for you?Rob Thorpe01/03/15 08:25 PM
                                          Avoiding ping pongMaynard Handley12/09/14 11:33 AM
                                            Avoiding ping pongPatrick Chase12/09/14 01:54 PM
                                              Avoiding ping pongMaynard Handley12/09/14 06:56 PM
                                      Avoiding ping pongSalvatore De Dominicis12/09/14 08:51 AM
                                        Avoiding ping pongPatrick Chase12/09/14 02:00 PM
                                      Avoiding ping pongook12/11/14 03:31 AM
                                      Avoiding ping pongArt Scott12/19/14 10:19 PM
                                        Avoiding ping pongEric Bron nli12/20/14 04:05 AM
                                      What about specialization?Troll?01/02/15 07:55 AM
                                        What about specialization?Ungo01/04/15 03:27 PM
                                      Avoiding ping pongfewwef01/05/15 08:16 PM
                                      Avoiding ping pongV.Krishn01/08/15 06:11 AM
                                    Avoiding ping pongGabriele Svelto12/08/14 07:32 PM
                                    Avoiding ping ponganon12/08/14 11:37 PM
                            Avoiding ping pongKonrad Schwarz12/10/14 06:23 AM
                              Avoiding ping pongLinus Torvalds12/10/14 11:56 AM
                          Object reference lockingDavid W12/08/14 11:36 PM
                            Object reference lockingPatrick Chase12/09/14 04:52 PM
                              Object reference lockingDavid W12/11/14 05:18 AM
                    ISA != interface for "most programmers"Paul A. Clayton12/04/14 03:34 PM
                      ISA != interface for "most programmers"rwessel12/04/14 07:50 PM
                  Interesting! (exporting hot lines/cache-aware ISA); "Please sir, I want some more" (NT)Paul A. Clayton12/04/14 02:26 PM
                  Avoiding ping pongMichael S12/06/14 03:48 PM
          ARMv8 getting atomic operationsLinus Torvalds12/04/14 12:05 PM
            LL/SC idiom recognition is not admitting RMW superiorityPaul A. Clayton12/04/14 02:34 PM
            ARMv8 getting atomic operationsanon12/04/14 10:17 PM
    ARMv8 getting atomic operationsPatrick Chase12/03/14 12:09 PM
  limited ordernksingh12/04/14 10:17 PM
    I didn't understand this either. (NT)Konrad Schwarz12/04/14 10:32 PM
    limited orderdmcq12/05/14 02:13 AM
    limited orderbakaneko12/05/14 09:11 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?