Brendan (btrotter.delete@this.gmail.com) on January 17, 2020 12:51 pm wrote:
> My main argument is that overcommit needs to be phased out because it has resulted in "too
> much software is bad and doesn't handle allocation failures gracefully". I'd be proving
> myself wrong if I could show codebases gracefully handling all allocation failures.

And you provided exactly zero evidence to back this assertion.

> So you're saying that using signals to avoid and/or recover from "OOM killer" is so easy that
> you couldn't do it, so you resorted to restoring processes that were killed by OOM instead?

Actually we're doing that mainly on Windows.

> Great; now code that explicitly checks for allocation failures and ensures the error is reported
> in a manner consistent with the way other errors are reported is the same as pretending that
> allocations never fail and letting OOM killer worry about reporting the problem.

Yes, it is. It removes the error checking from everywhere else because that's impossible to handle in a sane way.

> I didn't look at every single crash report. I looked at about 12 of them. 11 of them
> looked extremely similar; and one looked very different (mostly because it failed to
> find most symbols in the backtrace so a large part of the bottom of the page was red)
> so I ignored it thinking it was a outlier (possibly misattributed to OOM).
>
> > A quick check would have
> > revealed that machines in the >90% bracket are made up mostly of users with 2-, 3- or 4GiB of RAM.
>
> Yes, and? The amount of "RAM+swap" that the OS is willing to commit to providing (which
> is the amount FireFox exceeds) does not depend on the amount of RAM alone. Of course
> with a large amount of "memory that kernel committed to providing that isn't actually
> allocated (yet)" the amount of free physical RAM is also relatively unimportant.

You were the one that brought up the system memory usage, not me. The amount of physical RAM is important because on Linux and macOS you OOM only when you've not only exhausted all your physical RAM but you're thrashing your swap file hard. On Windows you run into an OOM much earlier when free RAM is actually available.

That's a flaw Windows has and macOS/Linux don't.

> That's easy enough:
>
> a) your statistics don't include "OOM killer terminated a completely unrelated process", causing
> an unknown number of incidences of resource starvation on Linux and MacOS to be uncounted.

Yes it does. In fact I'm overestimating the Linux OOMs because you can get SIGBUS crashes for other reasons beside the OOM killer.

> b) FireFox is not designed purely for "no overcommit" (and is designed for portability,
> supporting systems with and without overcommit), and is therefore more likely to
> overcommit and less likely to avoid crashing when it does overcommit.

Until two years ago I didn't even know that Windows wouldn't overcommit. I discovered it because I started looking at why we were OOM'ing so often and the data looked odd because most of the crashes still had free memory. So no, we didn't give a damn about this, let alone "designed" Firefox around it.

> c) FireFox attempts to cache a large amount of "previously used" data in an attempt to improve
> performance, so the resource consumption is continually increasing over time until it reaches
> some kind of threshold; but whatever "some kind of threshold" is is likely extremely wrong
> for Windows.

No, we routinely purge caches and intermediate structures. We've got even a special purging event that runs once per day to ensure stale data on very long-running instances doesn't stay around.

> Mozilla's workaround is "Restart Firefox" ( https://support.mozilla.org/en-US/kb/firefox-uses-too-much-memory-or-cpu-resources#w_restart-firefox ).

It's funny that you attribute this to Firefox directly. All allocations in Firefox are in response to what a given page requests. If a site leaks, or holds onto too much data at once we can't do anything about it. Would you blame Java and .NET virtual machines for using too much memory too when running large programs?

> Software I write never continually increases its consumption until its hogging all memory the OS is willing
> to commit to, never needs anything like "timer based polling of resource consumption" (that's susceptible
> to rapid changes in resource consumption by other processes while my process is idle), relies purely on
> handling allocation failures when they occur (and never crashing when allocation failures occur), tries
> to minimize the amount it expects OS to commit to providing (never using fork, never using multiple processes
> when threads will do, never memory mapping files as "read/write", etc), and (in general) is designed with
> the assumption that the OS supports multi-tasking (and there may be many other processes running that are
> competing for the same resources, hopefully using the same resource management tactics I am).
>
> So, no, applications I write never need to go to those lengths to prevent OOMs.

But you claimed that we didn't go to these lengths, that we just had "screwed up new()", when instead we behave better than most other applications even WRT other processes in the system. We even de-prioritize background processes to ensure we play nicely with whatever is in the foreground.