Brendan (btrotter.delete@this.gmail.com) on January 15, 2020 5:20 am wrote:
> No, you're assuming overcommit exists and its leading you to false assumptions.
>
> If an allocation fails and there is overcommit; yes, you're probably screwed regardless.
>
> If an allocation fails and there is no overcommit; then it's not a problem at all -
> you're able to do anything you like to handle that (without needing any notifications)
> without worrying about "OOM killer" preventing you from doing anything sane.

No, you assume that you can't detect the OOM killer killing you. You can, you just need a handler for SIGBUS. The only scenario in which you can't detect the OOM killer is IIRC when the kernel needs memory and then it will take it no matter what. I don't know how the Windows kernel deals with that but I doubt it will fall over and panic instead of killing a user process.

> No, it's easy to pre-allocate whatever you need to handle errors like this (e.g. during process
> startup) to ensure you don't need to allocate when you can't. Sometimes that's not even necessary
> - in my case (server with one thread per connection) I can output to log without allocating
> and then terminate the thread/connection (which also causes its "thread local data" to be
> freed and helps other threads/connections avoid future allocation failures).

OK, your assumption of "handling OOM situations correctly" is:

• You can recover from the failure of every allocation
• You will have error handling for every allocation

That's not possible unless your handling is "OMG we're out of memory, spit something on the log and abort". That is not what I call handling an OOM condition. We have plenty of fallible allocation points in Firefox where we can actually fully recover from an allocation failure and go on, but it's not possible to do that for every single allocation, especially if you don't control all your dependencies (and that's practically impossible nowadays).

That also assumes that all allocation points (including indirect ones) have proper checking. Because if they don't you'll have NULL pointers spreading through your data possibly causing a random crash at a later point where you can do absolutely nothing about it. You can't just assume that software has perfect error detection.

> Of course if you do have overcommit, then things like modifying data or writing to a log
> file could cause "commited to but not actually allocated" pages to be allocated (and trigger
> "OOM killer"); so trying to handle OOM conditions does become a "tip-toe through the minefield"
> nightmare scenario that you've falsely assumed applies to "without overcommit".

I'm not sure what you're talking about. The OOM killer will send you a signal which you can handle in a signal handler and you can even resume from it. The process doesn't disappear into thin air unless the kernel is running out of memory and then it's obviously better to kill a user process than the entire machine. As I pointed out the above applies to Windows as well. Also, you can control the OOM killer from user-space.

> https://stackoverflow.com/questions/45605806/out-of-memory-allocating-bytes
>
> Note: "failed to allocate" was correctly handled by logging/reporting the error (instead
> of letting a random and potentially unrelated process get terminated by "OOM killer").

LOL. GCC is just quitting at that point possibly leaving behind a half-written object file. I don't see how that's better than dying at a random point.

GCC is also not a really good example because it handles SIGBUS on Linux and when it does it tries to quit more gracefully (though not by much).

> You're being disingenuous by assuming a system without overcommit
> will be set up the same as a system without overcommit.
>
> If you have one system that overcommits, and another system that doesn't overcommit but is given
> adequate swap space to ensure that it can handle the exact same load as the first system; then
> the only difference will be that the first system has "OOM killer" and may terminate unrelated
> processes (which is incredibly idiotic) and the second system will be able to handle OOM in
> whatever manner the developers wanted to (which can be significantly less idiotic).
>
> The important thing is how OOM is handled, not how many cents
> worth of extra swap space are needed to avoid idiocy.

You're handwaving stuff away without presenting data to back your assumptions. I have crash data - from million of Firefox users - that shows over two thirds of OOM crashes on Windows happening on machines with plenty of physical memory available. That means that Windows lack of overcommit causes a very large amount of crashes that would simply not happen on a system with overcommit.