Brendan (btrotter.delete@this.gmail.com) on July 14, 2021 12:50 pm wrote:
> Hi,
>
> dmcq (dmcq.delete@this.fano.co.uk) on July 14, 2021 10:13 am wrote:
> > Actually I did write an operating system myself from scratch complete with multiple virtual
> > machines about thirty years ago. The most difficult error handling was dealing with memory
> > errors as we expected a couple a week to occur in the various boards using it.
>
> I apologize for completely changing the topic; but I've been researching/thinking about/"theorizing
> about" memory error tolerance (on and off) for about 10 years now - mostly revolving around
> the obvious "software ECC on top of paging" approach (as described in papers like this one:
> https://repository.lib.ncsu.edu/bitstream/handle/1840.20/33537/TR-2016-7.pdf ).
>
> Are there any interesting insights and/or details you'd be willing to share regarding your experiences?
>
> - Brendan

Nothing particularly wonderful. The memory had parity but not hamming - the problem would have been much less worrying with hamming but that's what saving money does to you. Tasks for the board could be discarded and restarted easily if it said there was an error which made things easier. On the other hand if it didn't recover in a halfway decent manner from a failure the host would be unable to restart the board after timing it out.

As to the handling, chunks of read only areas were covered by longitutinal sums and were checked incrementally in the watchdog timer and fixed if there there was only one error per chunk. Errors could also be detected whilst running or in IO. Read only errors would be fixed. If an error occurred in an unused area including parts of buffers which weren't full the error was just fixed. Otherwise errors in writable areas which could be ascribed to a particular VM caused an error to be returned for the task, otherwise the board said there was a general error and everything was restarted. The handler for the errors had a little bit duplicated and provided one got past an initial little part an error in either half would be handled by going down the other path. Stuck errors were handled using virtual memory. Details would be returned about where errors happened rather like a disk.