anon (anon.delete@this.anon.com) on July 12, 2015 3:42 am wrote:
>
> So all of that happens inside the core. Memory ordering instructions
> have to prevent these reorderings within the core.

No, they really don't.

Look, let's make this really simple. You have this code:

- store A to location X
- load B from location Y

and you want to move the load earlier, because loads matter from a performance standpoint, and stores don't and can be buffered.

Ignore any SMP issues for now. So imagine that you're running on a UP system, and there is no DMA or anything like that. In fact, ignore even caches. Imagine that the loads and stores go directly to memory, there are no caches, there are no other agents, there is absolutely nothing but you and the memory.

So by definition, any amount of memory barriers between the two operations don't matter. Agreed?

What do you do if you want that sequence to run as fast as possible?

Now, trivially - and traditionally - the thing to do is to just verify that there is no overlap between X and Y, and just do the load early. Agreed?

That actually takes care of a lot of things, and is reasonably simple. You can move lots of loads up much earlier, and life is relatively good. Let's call this "phase 1".

Now, let's say that you want to move the load up early, but the store address isn't known yet. What do you do now? You've taken all the low-hanging fruit, you're doing a reasonable job of OoO and maybe have a hundred+ instructions in flight at a time, and you notice that you still have a fair number of loads from addresses you know early, but you can't actually do yet because they are all interspersed with stores that you haven't the addresses for.

"That sounds unlikely" you say. You'd be wrong. Things like loads from constant areas etc are quite common, but the function prologues have all those silly stores to the stack to save registers etc, so you can't do those critical loads early. So you need to extend on your load re-ordering.

So what do you do? You know that those stores to the stack area are not going to interfere with the loads from the constant area, but how do you prove it? You don't. You set up some simple prediction mechanism, and you say "ok, I'll just assume that when I see a store to the stack with a constant offset, it's spilling registers, and it's not going to affect a load off a random non-stack pointer". So I can move the load up and do it early, and those stack spills won't screw with my critical load.

That sounds like an obvious thing to do, right? When you save your registers to the stack, you're basically going to be loading them the same way 99% of the time, and this means that the loads exposed by your OoO engine going into the next function call can be started early, despite all those pesky function prologue stores.

But you realize that "99% correct" is not quite good enough, so now when you dot he load early, you'll have to *check* that the load was correct. So you have to add support to your core to do speculative loads: do the load early, but leave some kind of marker in your memory subsystem that gets verified when it's time to actually retire the load. So *if* the stores to the stack overwrote the memory location you did the load from early, you notice when you're going to actually retire the load, and instead take a mispredict and re-do the load in order.

And you can tweak that load/store re-ordering prediction any way you want. Maybe you just start with that "stack vs non-stack" heuristic. But I think you'll agree that you might extend it to other scenarios (including some branch-prediction-like thing that remembers which load instructions shouldn't be re-ordered because they caused problems before). That obviously doesn't really change anything. Agreed?

See? None of this has anything to do with SMP so far, and all of this is simple and straightforward, right? Let's call this "leave a marker and do the load speculatively" behavior "phase 2".

And I think we can agree that obviously the CPU that does "phase 2" is the more intelligent one, and likely to be able to perform better. Yes, it's more complexity, but the memory subsystem is so important that that complexity is definitely worth it unless you're doing something really simple.

So, with "phase 1" and "phase 2" in mind, let's see what happens in SMP. We've ignored that entirely so far.

Now, "phase 1" really doesn't work at all in SMP. I think we all agree. You can't reorder a load at all across other stores you do - or even other loads - because you don't know what stores the other CPU's are doing. All in agreement?

In fact, let's introduce caches here. Remember: we didn't have SMP nor caches to worry about in our original model of "phase 1", but we might as well introduce caches now and see what it does.

In particular, you realize that caches fundamentally kind of act like moving that "load B from Y" earlier - the fact that the core cached location "Y" from an earlier situation means that the CPU basically did an early load. That wouldn't have mattered when you were the only owner, but we ignored it for simplicity, and now we realize that caches fundamentally act as that kind of "phase 1" reordering even in the absense of OoO and clever instruction-level re-ordering.

Agreed? With me so far?

So what do you do? You introduce memory barriers. They make sense even in strictly in-order designs, because as we noticed, caches and store buffer interactionsneed to be serialized too.

And this is where all those f*cking stupid "weak memory ordering" shit architectures are. They got to phase 1, and wanted to do SMP, and even if they weren't even OoO, they had realized that they needed memory barriers to synchronize the store buffer and the cache accesses. Maybe they weren't smart enough to realize that OoO was a good idea, but they had noticed that nonblocking caches were a basic requirement for any kind of performance, so you had multiple loads in flight even in an in-order core, never mind just the store buffer.

So obviously memory barriers are a good idea, and an absolute must. It's what synchronizes all those things.

But it's only "obvious" if you're a CPU designer who is busy sitting in a corner eating paste, and you left behind at that "phase 1" level.

Quite frankly, that "phase 1" argument was valid 20 years ago. Today it really is "paste eater" level crap.

Because look at what happens when you introduce SMP (and caches, I think we can agree that they are just another aspect of the whole "reorder memory accesses") in a "phase 2" design.

Really. Stop here, and think it through yourself. What does a "phase 2" design do in SMP. I'll leave a few empty lines for this concept to sink in.

.

.

.

So the problem with "phase 1" and SMP was that it checked the load against the other known accesses it reordered against, and that obviously only works when you know what those other accesses were - ie locally. "Phase 1" fundamentally doesn't work for SMP.

But "phase 2"? Ignore all those memory barrier ideas, because they were just the obvious solution for "phase 1", and they are crap, and they are wrong. Instead, go back to the original phase 2 model: we do the load early, leave a marker in the memory subsystem, and then just check the marker when we actually retire the load in-order (and I say "retire", but it's not necessarily the same as "instruction retirement", you could think of it as a separate in-order queue of "memory access retirements" which might or might not have anything to do with the actual instruction retirement).

And the thing is, that model works fine with caches and with SMP. There are no memory barriers needed. Memory barriers are just pointless crap. They don't add anything at all, they just distract you from the real solution.

So just look at that example of "do a load early" model: just do the load early, you marked it somewhere in the memory subsystem, and you added it to your memory access retirement queue. Now you just need to figure out if anybody did a store that invalidated the load.

And guess what? That's not so hard. If you did an early load, that means that you had to get the cacheline with the load data. Now, how do you figure out whether another store disturbed that data? Sure, you still have the same store buffer logic that you used fro UP for the local stores, but you also see the remote stores: they'd have to get the cacheline from you. So all your "marker in the memory subsystem" has to react to is that the cacheline it marked went away (and maybe the cacheline comes back, but that doesn't help - if it went away, it causes the marker to be "invalid").

See? No memory barriers. No nothing. Just that same model of "load early and mark".

And the important take-away from this is two-fold:

(1) notice how the smarter CPU core that didn't need the memory barriers didn't re-order memory operations less. No sirree! It's the smarter core, and it actually re-o5rders memory operations more aggressively than the stupid core that needed memory barriers.

In fact, please realize that the memory barrier model performs fewer re-orderings even when there are no memory barriers present - but it particularly sucks when you actually use memory barriers. The smart CPU "just works" and continues to re-order operations even when you need strict ordering - because in the (very unlikely) situation that you actually get a conflict, it will re-do things. The stupid memory barrier model will actually slow down (often to a crawl), because the memory barriers will limit its already limited re-ordering much further.

(2) weak memory ordering is an artifact of historical CPU design. It made sense in the exact same way that RISC made sense: it's the same notion of "do as well as you can, with the limitations of the time in place, don't waste any effort on anything 'clever'".

So you have that odd "dip" in the middle where weak memory ordering makes sense. In really old designs, you didn't have caches, and CPU's were in-order, so memory barriers and weak memory ordering was a non-issue. And once you get to a certain point of complexity in your memory pipe, weak ordering again makes no sense because you got sufficiently advanced tools to handle re-ordering that synchronizing things by serializing accesses is just stupid.

The good news is that a weak memory model can be strengthened. IBM could - if they chose to - just say "ok, starting with POWER9, all of those stupid barriers are no-ops, because we just do things right in the core".

The bad news is that the weak memory model people usually have their mental model clouded by their memory barriers, and they continue to claim that it improves performance, despite that clearly not being the case. It's the same way we saw people argue for in-order cores not that many years ago (with BS like "they are more power-efficient". No, they were just simple and stupid, and performed badly enough to negate any power advantage many times over).

(Oh, and a final small note: things aren't quite as black-and-white as "weak memory ordering or not". The x86 memory ordering isn't a total sequential consistency, and there are graduations and variations between "weak" and "strong". The alpha memory ordering is particularly weak and crappy, while the x86 and sparc TSO orderings are fairly strong but allow stores to be re-ordered after loads, which simplifies some things with fairly minimal disadvantages. Note the "after" part - it's actually important that stores can only be delayed but never done early!)

Linus