Hi,

dmcq (dmcq.delete@this.fano.co.uk) on July 14, 2021 4:27 pm wrote:
> Brendan (btrotter.delete@this.gmail.com) on July 14, 2021 12:50 pm wrote:
> > dmcq (dmcq.delete@this.fano.co.uk) on July 14, 2021 10:13 am wrote:
> > > Actually I did write an operating system myself from scratch complete with multiple virtual
> > > machines about thirty years ago. The most difficult error handling was dealing with memory
> > > errors as we expected a couple a week to occur in the various boards using it.
> >
> > I apologize for completely changing the topic; but I've been researching/thinking about/"theorizing
> > about" memory error tolerance (on and off) for about 10 years now - mostly revolving around
> > the obvious "software ECC on top of paging" approach (as described in papers like this one:
> > https://repository.lib.ncsu.edu/bitstream/handle/1840.20/33537/TR-2016-7.pdf ).
> >
> > Are there any interesting insights and/or details you'd be willing to share regarding your experiences?
> >
> > - Brendan
>
> Nothing particularly wonderful. The memory had parity but not hamming - the problem would have been much less
> worrying with hamming but that's what saving money does to you. Tasks for the board could be discarded and restarted
> easily if it said there was an error which made things easier. On the other hand if it didn't recover in a halfway
> decent manner from a failure the host would be unable to restart the board after timing it out.
>
> As to the handling, chunks of read only areas were covered by longitutinal sums and were checked incrementally
> in the watchdog timer and fixed if there there was only one error per chunk. Errors could also be
> detected whilst running or in IO. Read only errors would be fixed. If an error occurred in an unused
> area including parts of buffers which weren't full the error was just fixed. Otherwise errors in writable
> areas which could be ascribed to a particular VM caused an error to be returned for the task, otherwise
> the board said there was a general error and everything was restarted. The handler for the errors
> had a little bit duplicated and provided one got past an initial little part an error in either half
> would be handled by going down the other path. Stuck errors were handled using virtual memory. Details
> would be returned about where errors happened rather like a disk.

Thanks - it definitely sounds like an interesting piece of engineering (and made me think more about providing feedback higher levels could use to further manage uncorrected errors). :-)

- Brendan