Michael S (already5chosen.delete@this.yahoo.com) on March 5, 2021 4:06 am wrote:
> Linus Torvalds (torvalds.delete@this.linux-foundation.org) on March 4, 2021 8:47 pm wrote:
> >
> > All those MIS people want to know is which DIMM to replace, right?
> >
> > Linus
>
> Isn't it true not just for MIS (don't know what it means) but for me too, as workstation* user?

You do want to replace a known bad DIMM, yes.

But it's not always about a known bad DIMM at all.

So you want to do SO MUCH MORE than just that with ECC reporting.

So let's make a very concrete example: "Ok, one machine in the farm of a few hundred machines inexplicably hung, and was power-cycled by a hardware watchdog timer, and came back up, all happy again".

Ans notice how I say "a few hundred machines". If you have a farm of millions of machines, you probably end up having very special hardware to help log this, and honestly, you don't even care about a single machine - you start caring only once you start seeing big patterns ("these machines we bought in the same batch have statistically significant behavior from other ones").

But if you're a regular company, with some random internal set of machines, you probably do not have hugely specialized logging hardware outside of the trivial "system management ethernet network with virtual consoles and serial lines". Which help for the obvious problems, but are almost entirely useless for the "machine hung with no messages" kind of problems.

Sure, you can continue to use that machine once it came back up, but what you really want to do is to have some automated post-mortem, and so what you'd like to do is save off the old memory image so that you have some idea of what happened. Was it the PCI bus that hung up because of some glitch, was it software that deadlocked with interrupts disabled, what was going on?

So what you'd actually want to do is read the old memory contents as you bring it up again. Perhaps not all of it. Perhaps you just have a part of the memory set aside for some fast logging exactly so that in these situations you can at least see "what was the last thing the machine was doing".

In fact, you'd like your standard OS image to just do that for you automatically, so that you're not actually even going to need hugely specialized knowledge.

So you want to read all that memory, and it is going to give you ECC errors, because it hasn't been scrubbed and you cut power to it for a second (because (a) that's easy and (b) it's sometimes the only thing that really brings the machine back when the hardware is wedged).

So you get all those ECC reports, and you're going to mostly ignore them, but they still might contain useful data even for the non-scrubbed memory.

But wouldn't it be good to know which ones to ignore, and which ones not to ignore, because some of the ECC reports might be from the memory that you have scrubbed and is part of the newly booted image. Maybe that's literally the reason things went sideways?

This can be really hard on lots of hardware, because quite often the ECC error reporting is really really badly done as I outlined. In this case, you don't care about which memory channel and DIMM and rank it was - because that is actually an almost completely random mapping of the actual virtual/physical address that the CPU instruction stream is using (because of interleaving both on a NUMA level and on a chip level, because of address bit remapping by hardware both on the motherboard and inside the CPU chip etc etc).

And also, please take note of that "you want the standard OS image to do this all for you". This needs to be standard and architected. Not some kind of "on this particular hardware it works like X, and you look up the ECC details using these model-specific registers, and then translate them using this other machine-specific register, and then do a third translation using the BIOS tables described by this odd corner of the ACPI spec that nobody ever really tested and verified since the spec is 2500 pages long and there are so many other odd corners".

See what I'm saying? ECC is just so much more than just "oh, your DIMM is bad, please replace the DIMM that is in slot 5".

It's about "maybe I intentionally didn't scrub part of the memory, and it would be lovely to know how much of it actually I can trust even though it wasn't getting refreshed for a second, but honestly, I'll happily take even corrupted data and look for patterns".

It's about "maybe I'm under a rowhammer type attack, and I really want to know where the accesses are coming from that cause these ECC bit flips, and what the CPU physical and virtual addresses were - I don't care one whit which DRAM chip it was!".

It's about "maybe it's not one DIMM that is going bad, maybe I have some odd high background radiation levels that I didn't even think of, because it turns out I put my server in a basement and it turns out it has exceptionally high radon levels".

See?

But in reality, ECC error reporiting is a huge mess, and usually horribly badly done. It's almost never architected, so it's a "this chip family does this". It's very seldom well-designed, so it's often hard to impossible to do certain mapping of "I got an ECC error" to the source (usually people do make sure that you can figure out at least which DIMM slot it is in, so that the "replace this DIMM" at least works, but even that is often some black magic).

And yes, I'll also very happily admit that systems like the Linux kernel don't necessarily do as good of a job as we could do. Because 99% of developers have no access to the facilities in the first place, can't really test it very well, and because it's not architected, it's not even "this is how you do it on x86" - it's "on this family of Xeon CPU's with this setup, this is how you do it".

Is it any wonder that I'm frustrated with it?

Linus