Brendan (btrotter.delete@this.gmail.com) on January 14, 2020 1:30 pm wrote:
> Jukka Larja (roskakori2006.delete@this.gmail.com) on January 14, 2020 10:13 am wrote:
>
> > Not untestable, but in practice hardly ever tested, because OOMs practically never happen.
> >
> > Every game console requires us to handle out of disk space
> > (and other user data write errors) in some sensible
> > way.
>
> If someone ignored the error during write then (later) crashed while trying to load
> data that wasn't correctly written, would you assume the software is stupid? Why is
> it stupid for one resource (disk) and "normal" for another resource (memory)?

Handling doesn't mean just not crashing. It means showing particular errors based on rules that depend on what exactly the error was and what is the error state of the system (e.g. should multiple errors about the same thing be shown individually or just as a single popup). There are also rules about which errors may be reported to user, which errors must be reported using API functions only and so on.

There are no such rules about OOM, of course, because the basic rule "never crash" takes care of that. My point is mostly that compared to trying to handle OOM, handling the save data access errors is trivial, yet it's still a huge thing compared to actual save-data-access-in-errorless-case.

To answer the question, handling OOM is stupid, because it is so much easier to avoid it. That's software and system dependent of course, so if in your particular case it is easy to handle and OOMs happen all the time, then it isn't stupid.

In some system it may be easier (like a game console, where amount of available memory is always the same) than on others (normal PC, where amount of memory varies, and amount of available memory varies even more, in real time). And in some software being able to hold some huge cache when there's a lot of available memory may be such an advantage that it makes sense to try to live as close to OOM as possible.

Every single one of the apps with such caches I've seen (various audio software suites, CD burning software, both in-house and third party build tools), have managed them badly, usually just allowing to set the minimum and maximum. Basically leaving it to user. And frankly, I don't think I could come up with any idea how to do better. By the time things slide to OOM it's far too late to back out.

> > Even with ready made tools to simulate the problem, it's still a bitch to test, because it's very
> > much out of the normal execution path. It never gets tested, except when someone spends time to do just
> > that. Devkits never run out of disk space in any normal use. In practice, it's easier to get saving the
> > data right than it is to handle the case when saving fails. Not because it's actually more complicated,
> > but because the non-problematic saving gets tested couple of orders of magnitude more often.
>
> Seems like it'd be trivial to pretend a write returned an error when it didn't.

Just looking at the specs that would be my first thought too. My practical experience says otherwise.

The problem is that on no platform you can actually write anything to disk in main thread (On PS4, for example, mounting and unmounting save data takes about 200 ms each and you can only hold it mounted for about five seconds. It doesn't really matter, since HDD seek time would be too long in any case). So you setup the access and go on and then on some random moment the thread doing the access will tell you that it went wrong, and you have to somehow handle it. And you have no idea in what state the whole program actually is.

So you come up with something, preverrably trivial to avoid any problems, test it with some trivial "every access fails" test, commit it and let QA know that save data errors should now be handled. So far on every platform QA has come up with something that broke the trivial system. Often this involved incredibly accurate timing (going to that menu, loading a level, then quitting about two seconds later etc.), or some other special circumstances (e.g. all save slots and their backups filled).

In modern game engines, common problems have annoying habit of turning to threading and timing problems.

> The problem is that allowing overcommit prevents all software from handling OOM, not just software that
> doesn't handle OOM. You are saying "I don't handle OOM, therefore nobody should be allowed to handle OOM".

Most of those special apps could probably code around that just by touching all the memory they allocate.

> Note: If someone calls "mmap()" and doesn't bother to check if it returned -1, then I do consider them
> incompetent. At a minimum I'd expect some kind of "fprintf(stderr, "Failed to mmap, reason: %sn", strerror(errno)
> );" where OOM is at least treated the same as any other error condition and not ignored.

I mostly agree on that, though in case of memory allocations, we have our own memory manager which will report the error, then return null. It's pointless to check the return value, since error has already been reported and if we were trying to handle the OOM, we'd do it in the memory manager.

For pretty much any use of any third party API, I feel allowed to yell at anyone who doesn't check the return values though. But that's just because you don't know what sort of error, if any, will be reported. It doesn't necessarily even mean that we try to avoid crashing.

-JLarja