Travis (travis.downs.delete@this.gmail.com) on August 19, 2018 11:36 am wrote:
> Maynard Handley (name99.delete@this.name99.org) on August 18, 2018 2:42 pm wrote:
>
> > compressed pages), or remote atomics (to speed up common atomic operations like ARC increments);
>
> We've been over this before, but remote atomics make no sense for the vast majority of
> code. A "remote atomic" necessarily has a latency of whatever shared thing is remote and
> doing the increment, so in the range of 10s of clocks for something like L3, or in the
> 100s of clocks if it happens in memory (or a bit less if in the memory controller).
>
> Existing atomic operations are almost totally local operations, that work
> at "core speed", and don't require any coherence traffic at all (except when
> another core wants the line, just applies to non-atomics as well).
>
> The overwhelming majority of the time when something line is operated on atomically, the
> last core to use that line was the same core. Making this case fast is really important.
>
> Remote atomics is like throwing out the best case which almost always happens in favor of always worst-case
> behavior, just to make that worst-case behavior a little better. Now every atomic access in your
> program has to go out over long-latency, power sucking busses, and the core has to wait forever to
> get the result back (and it almost always does need the result, as in your ARC example).
>
> It just makes no sense as an implementation of existing atomics.
>
> Now there are some specific sharing patterns where it may work, but it would have to be a new instruction.

I think there's a good counter-example to your point, which is the Itanium McKinley implementation of fetchadd. If the line is in L1, the CPU does it. If it's not, the core tells the L2 to do it. Obviously no-one's citing Itanium as a good example of much these days, but I never heard anyone complain about its implementation of atomics.

Let's talk about something that might make sense. A lock is typically on a cacheline which contains other data that is also being operated on. If the lock is uncontended (either no CPU has the cacheline or another CPU has the cacheline in an unlocked state), you want to bring in the cacheline. If the lock is contended, the last thing you want is to steal the cacheline from that CPU; at the very least it wants to access the cacheline again to release the lock, and chances are it's operating on other data in that cacheline.

So _if_ we have a way to indicate to the CPU that "this op is a lock acquire", then it might make sense to have a mechanism in the inter-core protocol to have the L2 cache steal the cacheline from the CPU, perform the operation, then _depending on the result of the operation_ either return the cacheline or report back to the requester the result of the operation.

It's fiendishly complicated. It'd almost be better to have a way to transmit a series of operations and have the remote CPU execute them, because there isn't just one way to acquire a lock.

It's possible there are cases like "update global counter" that might be simpler to support than "acquire lock without troubling remote CPU too much".

Maynard's ideas are, as usual, poorly thought through.