Hi,

Gabriele Svelto (gabriele.svelto.delete@this.gmail.com) on January 19, 2020 11:55 pm wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 18, 2020 2:15 pm wrote:
> > Fine. I hereby formally submit FireFox as an example of a piece of software that doesn't handle all
> > allocation failures gracefully. My evidence includes hundreds of pages of crash reports despite developers
> > putting an overwhelming amount of effort into failing to solve problems that were caused by:
> >
> > a) overcommit masking symptoms of idiotic "continually increasing
> > without limit resource consumption" design decisions
> >
> > b) overcommit creating an attitude of "can't justify the hassle of dealing with allocation
> > failures' in some developers, leading to parts of the software being shit (e.g. not
> > passing failure up the call chain to a point where the user can be informed "Sorry,
> > not enough memory to do that" without screwing everything else up/crashing).
>
> Alright, now I see your point: I'm holding it wrong.
>
> > It's entirely possible, you just refuse to try.
> >
> > Lets's look at a piece of "my" evidence (the first crash log in the list): https://crash-stats.mozilla.com/report/index/ed62286b-ccd3-4d3f-b8d3-5d4a00200108
> >
> > The bottom of the page shows about 20 places that could have caught a "std::bad_alloc";
> > where you are able to choose any of those ~20 places (wherever is most convenient)
> > to handle the allocation failure instead of crashing.
>
> See the name of the class at the top of the stack? It's called AutoEnterOOMUnsafeRegion because it's
> used by the GC to handle parts of its run where it can't recover from an OOM - it usually can but
> it's not always possible. A compacting generational GC sometimes need to allocate memory to be able
> to make forward progress and the heap is in such a state that a failure is not recoverable.

You mean; sometimes a GC has to pre-allocate memory at startup to ensure it has the memory it needs to make forward progress?

> Also, funky datapoint: there's a memory report attached to that crash report and it indicates
> that the resident set is 88,61MiB. Does that fit your definition of a resource hog? It's
> also a perfect example of the limitations of Windows: with 271 MiB free physical memory
> it cannot satisfy a 256 bytes allocation for a process that's using 88 MiB.

Who cares? Resident set size is irrelevant when you're asking OS to commit to too much. E.g. a simple "malloc(SIZE_MAX);" in a loop would be enough to trigger OOM on every 64-bit OS (unless it allows infinite overcommit) with an insignificant resident set.

> > Look at https://hg.mozilla.org/releases/mozilla-release/annotate/501aef7fe1d9622236600a7e53843d40d163a123/ipc/glue/MessagePump.cpp#l238
> >
> > That line of code is handling errors already!
> >
> > Where is the most convenient place to handle the error? I don't know (I'm not spending several
> > months getting familiar with your codebase and it's very likely that it's not quite as simple
> > as slapping a "try/catch" somewhere), but don't try to pretend it's impossible.
>
> If it were at all possible would we have developed an entire new language to deal with that?

How does Rust deal with the issue? Searches lead me to believe it was a joke for script kiddies that just crashes, but is "slowly evolving" in an attempt to fix its problem (e.g. https://github.com/rust-lang/rust/issues/46865 ).

Note that (as far as I know), Rust was created to avoid "pointer safety issues" in C (e.g. buffer overflows, etc) and memory management issues that are not related to OOM (e.g. use after free); partly by adopting the stupidity of garbage collection (and then later realizing garbage collection is stupid and abandoning it).

> > Sure; and vehicle manufacturers should avoid this "flaw" by installing petrol/gas gauges that lie
> > and tell you it can provide the resources to reach the next town causing you to end up stranded on
> > the highway between towns; and banks should avoid this "flaw" by telling customers they have lots
> > of money in their account causing customers to go bankrupt because they bought things they can't afford;
> > and restaurants should avoid this "flaw" by lying and tell customers they have plenty of meat to sell
> > and then when they find out they don't they should randomly select a customer to butcher.
>
> I think there must be a variant of Godwin's law for making car analogies in IT discussion threads.
>
> > If a site leaks or holds onto too much data, a browser can tell the user "Sorry, this one
> > page had to be closed (or maybe frozen/paused) because it leaks or holds onto too much data
> > (and the OS can't provide it, and there's no previously cached data to discard)".
>
> You just described exactly how Firefox behaves in that scenario.

Wait, maybe you're partially right.

If FireFox creates a whole new process for every separate web page and every separate piece of javascript (so that if one process crashes due to OOM it only effects that one piece and not the whole app); then yes, in theory FireFox might be handling OOM properly.

However, in that case it would basically be a malicious/trojan fork bomb designed to trick the OS into committing everything it can, and causing everything to fail. Every time it creates a new process it asks the OS to commit to providing memory for a new copy of the entire program's modifiable pages until the system as a whole starves. Of course it would also make it difficult for one piece (in one process) to ask the rest of the pieces (in other processes) to free up memory used for caches, etc (to avoid failure); so even in the "fork bomb" case it's likely that it's not handling OOM as well as it could.

> > No; but I would blame them if they crash because they've failed to handle allocation failures.
>
> OK, I gather from your arguments that this is an acceptable way of dealing with OOM crashes:
>
> type* foo;
> if (!(foo = malloc(sizeof(type)))) {
>   fprintf(stderr, "OMG, we're out of memory!");
>   abort();
> }
>

No; that's just the bare minimum handling possible, which can only be acceptable if there's no better way it could've been handled. For something like GCC (where it can't continue without the memory, doesn't have any cached data it can discard, and can't sacrifice some work to allow other work to succeed) the only other alternative is to wait in the hope other processes release memory; but that can't work in some very common use cases (e.g. "make -j" leading to multiple separate copies of GCC all waiting for each other to finish).

> But Firefox - which does precisely that and even manages to gather
> a minidump when that happens - is not handling OOMs correctly?
>
> > I'm not sure if/where I said FireFox doesn't go to these lengths, but it sounds like something
> > I might've said before I found out more about Firefox; so if I was wrong and should've said
> > something more like "FireFox is a bad because it does go to extraordinary lengths to prevent
> > OOM that are proven to be ineffective by 100+ pages of crash reports" then I apologize.
>
> Ah, but our Linux and macOS builds doesn't seem to be affected by
> that particular problem. I wonder what's different on Windows.

Previously you said the crash logs include crashes from Linux and MacOS (and that crashes on Windows are 20 times as likely).

Note that my current suspicion is that FireFox partly does things that are stupid (expects OS to commit to far too much, causing it to crash more on Windows because Windows is better at defending itself; but also causing crashing on typical Linux and MacOS systems that, for default settings, support some overcommit but do not support "infinite overcommit"); and that FireFox partly doesn't handle OOM properly in places that it can; and that you've been "squirming" (changing your argument from one side to the other) to pretend that it's not suffering from either problem when it's suffering from both.

- Brendan