Hi,

Linus; the reality is that most (not all) of the problems you're worried about are caused by user-space being designed for the Linux kernel (and other *nix kernels, and everything else that evolved from festering 1960s ideas) - software using fork() when it could be using something else (e.g. posix_spawn()), software using processes when they could been using threads, software not caring about how much it asks kernel to commit to and doing nothing to remain reasonable, software not caring much about handling OOM because overcommit is working against it, etc.

In other words; (speaking about "general purpose OS" and not "bare kernel without user-space") you're using the consequences of poor decisions in the past to justify the continuation of poor decisions in the future; with the result that the OS as a whole (user-space) is perpetually prevented from improving.

Linus Torvalds (torvalds.delete@this.linux-foundation.org) on January 15, 2020 11:30 am wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 14, 2020 12:40 pm wrote:
> > Sure, with an adequate amount of swap space those crashes simply wouldn't have happened.
>
> It's easy to say "don't overcommit, just have enough swap space".
>
> But it's wrong.
>
> The thing is, it's easy to claim that overcommit is bad, but reality is often very
> very different. It's often horribly, horribly bad not to allow overcommit.
>
> This is one of those places where some people think the world is black-and-white,
> and those people are just simply and utterly wrong. If you think that overcommit
> is bad, you simply don't know what you are talking about.
>
> Overcommitting can be bad in some very specific circumstances. If you're doing hard realtime, or doing something
> truly safety-critical, and lives depend on it and nothing else matters, then yes, overcommit is bad.
>
> But very few problems in engineering are of the kind where "nothing else matters".
>
> Because engineering is about reality. And in reality, a lot of loads could end up using
> a lot more memory than they actually do. In reality, you have loads and loads of memory
> that ends up being shared, and committing to the theoretical case where all of them might
> need to become private copies is just a completely stupid waste of resources.
>
> Resources that cost real money (or real opportunity) in reality.
>
> In reality, a lot of memory ends up being stuff that you can throw out and re-generate even
> without swap, because it's a private file mapping that has never been written to - even if
> the process could have written to it. Maybe 5% of the mapped pages actually did get written
> to, and it's all mapped read-write because writing isn't wrong - it's just not common.
>
> Do you commit to the "this could happen" case, or the "this is what seems to be actually happening" case?

You commit to "randomly terminating an innocent process on a whim with no practical warning is completely and utterly batshit insane".

You commit to "a process being unable to reduce its memory consumption to allow itself (or other processes) to continue successfully is completely and utterly batshit insane".

You commit to "a process being unable to cancel some work to prevent other work from being disrupted (e.g. drop one TCP/IP connection and not others) is completely and utterly batshit insane".

You commit to "a process not even being able to report "Oops, mmap() returned ENOMEM" back to the user in a manner that is consistent with the way the process reports all other errors is completely and utterly batshit insane".

You commit to "an environment that uses garbage collection (e.g. Java VM) being prevented from collecting its garbage in response to (global?) low memory conditions is completely and utterly batshit insane".

You commit to "any extra swap space needed (and any performance differences) are insignificant compared to avoiding behavior that is completely and utterly batshit insane".

> And in reality, diskspace often isn't as cheap as people claim. You end up having other concerns, and
> while it's true that disk is much cheaper than RAM, it's not true that that means that you should just
> say "use 10x the diskspace of ram for swapspace, just in case", when in reality you don't need it.

In reality, you probably need it just to hibernate properly and/or to give you some leeway for abnormal/unexpected temporary usage spikes.

> There are also situations where people overcommit enormously. They are happily not nearly
> as common as they used to be: back in the bad old days of original Fortran, you often had absolutely
> enormous static allocations because your Fortran environment didn't have any real dynamic
> memory allocation capability, so you just sized your arrays for the worst case.
>
> Yes, F77 is finally dead, dead, dead. I hope. But it lived on for a long time, and
> people used programs written for it for a long long time. And having one huge static
> array and just using the portion you needed was the thing people did. Really.
>
> So if you then you run a much smaller problem on your desktop PC, you are really happy that the system
> is not being stupid and saying "you can't do that, because you don't have enough swap space".
>
> See? It's not black-and-white. Sometimes overcommit is a bad idea.
> And sometimes not overcommitting is an equally bad idea.

And sometimes being indecisive and leaving software developers no way to design software to suit an OS (because maybe the end user's system has lots of overcommit, and maybe the end user's system only some overcommit, and maybe the end user's system has no overcommit) is a bad idea.

> End result: a good approach tends to be to "overcommit gently" by default, but
> allow people to set other policies (including that "overcommit wildly" for those
> old Fortan cases, but also "don't overcommit") if they have specific needs.
>
> Because thinking the world is black-and-white in my opinion means that you shouldn't do engineering.
> Because you have a fundamentally incorrect world-view. There are basically zero real-world problems
> that have black-and-white answers. I think you'll find that even in "safety-critical" systems
> where peoples lives are at stake, you'll find that it's not really about "black-and-white", but
> often about "let's make the safety ranges 2-10x what we think they need to be".

Yes; the good approach for a kernel that is used by everything from tiny embedded system to huge servers is to allow the policy to be configured at compile time (and not to have a plethora of knobs and jiggly bits that either don't solve the problem or will be misconfigured by normal users).

However I'm talking about an OS, not a kernel. For a general purpose OS (e.g. Ubuntu) overcommit (and designing the user-space for overcommit) is silly (and every "for this specific process overcommit is needed" scenario you can think of is just "that specific process shouldn't have been, and probably wouldn't have been, designed like that to begin with" in disguise).

- Brendan