zArchJon (Anon.delete@this.anon.com) on March 4, 2021 6:39 am wrote:
> Ganon (anon.delete@this.gmail.com) on March 3, 2021 9:05 am wrote:
> > A recent pair of papers from facebook emphasized the importance of checksum protection even
> > within a single process:
> >
> > Facebook’s Tectonic Filesystem:Efficiency from Exascale
> > https://www.usenix.org/system/files/fast21-pan.pdf
> >
> > "
> > At Tectonic’s scale, with thousands of machines reading and writing a large amount of data every day,
> > in-memory data corruption is a regular occurrence, a phenomenon observed in other large-scale systems
> > [12,27]. We address this by enforcing checksum checks within and between process boundaries.
> > "
> >
> > and
> >
> > Evolution of Development Priorities in Key-value Stores
> > Serving Large-scale Applications:The RocksDB Experience
> > https://www.usenix.org/system/files/fast21-dong.pdf
> >
> > "
> > 11. CPU/memory corruption does happen, though very rarely,
> > and sometimes cannot be handled by data replication. (§5)
> >
> > 12.Integrity protection must cover the entire system in order to prevent corrupted data (e.g.,
> > caused by bitflips in CPU/memory) from being exposed to clients or other replicas; detecting
> > corruption only when the data is at rest or being sent over the wire is insufficient. (§5)
> > "
> >
> >
> > -----
> > Checksums & similar protect the data but what about the code (instructions)? The total data footprint
> > of instructions is smaller so the bitflips are less likely in practice there. Does hw take special
> > measures to protect instructions from corruption (more than it does for data)? What sw measures
> > make sense to protect instructions (assuming we need to care about this as well)?
>
> Along with legacy software, this is one reason people pay the extra price for IBM Z systems.
> All of the memory is protected with RAIM (Redundant Array of Independent Memory) which
> not only allows for an entire DIMM failure, but also provides robust error detection and
> correction on DIMMs. See: https://www.ibm.com/community/z/wp-content/uploads/sites/14/2020/04/sysdevblog-4cde-ibm2520zenterprise2520raim.pdf
> For more technical background.
>
> Along with protecting the memory, all of the caches and directories are protected and the CPUs
> even have robust error detection and transparent retry mechanisms when a soft-error is detected.
> And transparent CPU sparing when the error does not go away after a threshold of retries.
>
> Even access to remote disks is protected using additional CRCs in the FICON layer
> on top of Fiber Channel to detect bit errors above the physical protocol layer.
>

Yep they go in for checks in depth. I've wondered how they check for soft errors in the CPU. One could do parity or modulo checks for arithmetic opertions but one would need to duplicate logic operations I think. And of course all the control would need checking too. Triple redundancy checks is the simplest way round that I can see but I don't think they are doing that. I'd be inclined to just run the CPUs every so often in a test mode with various extremes of the clock and power and say that's fine for the next while if they pass.