Hi,

Gabriele Svelto (gabriele.svelto.delete@this.gmail.com) on January 16, 2020 2:13 pm wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 16, 2020 6:16 am wrote:
> > Oh, that sounds easy. [...]
>
> You still haven't shown me a codebase gracefully handling all
> allocation failures. Maybe it's not as easy as you claim.

My main argument is that overcommit needs to be phased out because it has resulted in "too much software is bad and doesn't handle allocation failures gracefully". I'd be proving myself wrong if I could show codebases gracefully handling all allocation failures.

> > You control which dependencies you depend on; and if you choose to depend on a hideous design failure
> > (e.g. that has functions that need to allocate memory but don't tell you they failed and force you
> > to assume something succeeded when it didn't) then that is your fault for making bad choices.
>
> Do you control Windows system libraries? Are they bug-free and always infallibly
> return error conditions? What about code coming from graphics drivers which
> you will encounter if you need to do anything which involves graphics?

Is everything always perfect? No. Does that mean nobody should try to improve things they do control because of things they don't control? Also no.

> > Um, what?
> >
> > If there's a library function that always returns "success"
> > (but the docs don't say that success is guaranteed
> > in all future versions of the library) and another programmer calls it without checking the return value
> > (that is always "success"), then I tell the other programmer they're incompetent and refuse to work with
> > that programmer ever again (because a future version of that library might return something else).
>
> So you're basically working alone or do you really believe that no errors ever slip through?
> We've designed Rust to make error handling both mandatory and easy precisely because forgetting
> error handling - or getting it wrong - happens all the time in practice.
>
> > Have you ever tried to use signals before? You get a signal while your in the middle of
> > who-knows-what; then what? Cause a deadlock while trying to acquire a lock you need to free
> > memory you were using to cache something? Inform a thread that it needs to terminate to
> > save other threads and wait for it to also receive SIGBUS and cause more problems?
>
> Yes, in fact I've written complex code running in signal-handlers across three platforms
> (Linux, macOS and Windows). We use those to gather data about crashed processes for
> reporting and recover. In fact we just landed a patch that will make Firefox front-end
> capable of automatically restoring processes that crashed due to an OOM.

So you're saying that using signals to avoid and/or recover from "OOM killer" is so easy that you couldn't do it, so you resorted to restoring processes that were killed by OOM instead?

> > Yes; GCC was just the easiest example of "bare minimum handling of allocation failures".
>
> No it doesn't. It just aborts instead of returning NULL so all the code in it can
> pretend that allocations never fail. It's doing the opposite of what you claim.

Great; now code that explicitly checks for allocation failures and ensures the error is reported in a manner consistent with the way other errors are reported is the same as pretending that allocations never fail and letting OOM killer worry about reporting the problem.

> > Heh, no. What you have is proof that a well known resource
> > hog is actually a resource hog. Look at the crash
> > reports - they're all saying "90%+ of system memory used, available page file size reduced to zero".
>
> Only 35% of those crash reports have >= 90% system memory usage. Not all of them. Over 10% have less
> than 50% memory usage. You clearly didn't look at the data. It's also disingenuous to look at system
> memory usage without correlating it to the total amount of physical memory.

I didn't look at every single crash report. I looked at about 12 of them. 11 of them looked extremely similar; and one looked very different (mostly because it failed to find most symbols in the backtrace so a large part of the bottom of the page was red) so I ignored it thinking it was a outlier (possibly misattributed to OOM).

> A quick check would have
> revealed that machines in the >90% bracket are made up mostly of users with 2-, 3- or 4GiB of RAM.

Yes, and? The amount of "RAM+swap" that the OS is willing to commit to providing (which is the amount FireFox exceeds) does not depend on the amount of RAM alone. Of course with a large amount of "memory that kernel committed to providing that isn't actually allocated (yet)" the amount of free physical RAM is also relatively unimportant.

> > Tell me,
> > how do you think Windows is supposed to hibernate properly
> > when you've gobbled 100% of RAM and 100% of disk?
> > The only problem I see here is that Windows isn't using per-process quotas to stop your memory leak sooner.
>
> How do you explain that OOM crashes are over 20 times more likely on Windows
> than on Linux and macOS? Isn't Firefox a memory hog on Linux and macOS too?

That's easy enough:

a) your statistics don't include "OOM killer terminated a completely unrelated process", causing an unknown number of incidences of resource starvation on Linux and MacOS to be uncounted.

b) FireFox is not designed purely for "no overcommit" (and is designed for portability, supporting systems with and without overcommit), and is therefore more likely to overcommit and less likely to avoid crashing when it does overcommit.

c) FireFox attempts to cache a large amount of "previously used" data in an attempt to improve performance, so the resource consumption is continually increasing over time until it reaches some kind of threshold; but whatever "some kind of threshold" is is likely extremely wrong for Windows. Mozilla's workaround is "Restart Firefox" ( https://support.mozilla.org/en-US/kb/firefox-uses-too-much-memory-or-cpu-resources#w_restart-firefox ).

Of course you would have a much better idea of why FireFox crashes (instead of handling allocation failures more gracefully - dropping cached data, politely telling user "sorry, not enough resources to do that", etc) than I ever will; and I would expect that at least some of the problem can be attributed to "necessary complexity vs. developer time" (e.g. in terms of "scope creep since the HTML 1.0" the amount of things a modern web browser has to worry about is daunting).

> > As far as I can tell (I'm "not great" at C++ and only looked at small scraps of a large project) the
> > reason they're crashes (and not "reductions in amount of data cached for later" or "dialog boxes telling
> > user one of their tabs needs to be closed" or anything else that could be considered acceptable) is that
> > FireFox's screws up "new()" so that (despite the "noexcept(false)") it fails to throw an exception when
> > there's OOM (preventing the caller, who has the knowledge needed to figure out the best way to handle
> > OOM and/or can send the error up the call stack to whatever is most suited to handling the OOM, from
> > being involved) and just ends up at an incredibly stupid "We might proceed to a stage 2 in which an attempt
> > is made to reclaim memory" comment in unfinished code that will never attempt to reclaim memory (because
> > actually trying to reclaim memory would be horrifically complex when you don't have any of the caller's
> > knowledge about the current context - e.g. which mutexes are currently held, etc).
>
> It's quite the opposite. Firefox tries very hard to free memory when it's approaching an OOM. I wrote
> the code that detects and responds to low memory conditions on Windows. It runs garbage collections,
> prunes caches, flushes buffers to save memory. I even wrote code that shuts down unneeded processes
> in response to that, to shrink memory usage even further. Heck, I wrote code that forces our custom
> allocator to aggressively give back pages it doesn't need to the operating system.
>
> Now tell me, do the applications you wrote go to these lengths to actually detect and
> prevent OOMs? Because if they don't you're being comically wrong about "Firefox screwing
> up new()". And it's even sillier considering that we have explicitly fallible allocators
> so that we can recover gracefully from failures in places where it's possible.

Software I write never continually increases its consumption until its hogging all memory the OS is willing to commit to, never needs anything like "timer based polling of resource consumption" (that's susceptible to rapid changes in resource consumption by other processes while my process is idle), relies purely on handling allocation failures when they occur (and never crashing when allocation failures occur), tries to minimize the amount it expects OS to commit to providing (never using fork, never using multiple processes when threads will do, never memory mapping files as "read/write", etc), and (in general) is designed with the assumption that the OS supports multi-tasking (and there may be many other processes running that are competing for the same resources, hopefully using the same resource management tactics I am).

So, no, applications I write never need to go to those lengths to prevent OOMs.

- Brendan