Hi,

Anne O. Nymous (not.delete@this.real.address) on January 13, 2020 12:22 pm wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 12:02 pm wrote:
> > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 8:43 am wrote:
> > > Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 7:21 am wrote:
> > > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 6:41 am wrote:
> > > > > Brendan (btrotter.delete@this.gmail.com) on January 12, 2020 9:19 pm wrote:
> > > > > > anon (anon.delete@this.b.c) on January 12, 2020 11:30 am wrote:
> > > > > > >
> > > > > > > Even if you write your own code extremely carefully, unless you a programming in an embedded context
> > > > > > > where you wrote all the code running on the platform, you can be pretty damn certain there is a
> > > > > > > lot of other parts in your system that are not able to gracefully handle OOM. There are just no
> > > > > > > server or desktop platforms in widespread use that can deal with OOM. It's entirely possible that
> > > > > > > your program sucks up all the memory, and then OOM is triggered by some crucial background service
> > > > > > > that nothing can live without, and which does not have viable alternatives to allocating.
> > > > > > >
> > > > > > > It's because of this that the sane option for desktop or server is to just ignore OOM and
> > > > > > > pretend it doesn't exist. Just hardening your own software is the worst kind of idiocy
> > > > > > > -- it's a massive waste of time, and it doesn't actually protect you from anything.
> > > > > >
> > > > > > This kind of thinking is the reason why the world sucks ("Some software is shit, therefore all software
> > > > > > should be forced to be shit forever and nobody should ever try to make software better").
> > > > >
> > > > > This assumes there's an easy way to "make software better". Sure, if during large parallel
> > > > > compile system runs out of memory, the compile can be restarted with less parallelism. And
> > > > > when our custom game asset builder runs out of memory, just restart from some previous known
> > > > > good state. Oh wait, that already happens when we just hit "Build" button again (or someone
> > > > > makes an SVN commit and build system automatically start making a new build).
> > > > >
> > > > > OOMs are practically never a problem for us. Running with not-quite-enough-physical-memory is sometimes
> > > > > a problem, but gets handled nicely by virtual memory. On average, we have plenty of RAM. Sometimes
> > > > > various independent processes with large memory footprint just happen to run at the same time.
> > > > >
> > > > > If I had to come up with some manual system to handle the problem, I don't really see what I could
> > > > > do to improve it. There could be some heuristics about available memory that would affect how much
> > > > > parallel processes are launched. The problem is coming up with good heuristics. I have no idea how
> > > > > much the shader compiler will need this time, or if it needs to run at all. There's hundreds of variables
> > > > > to consider, each only making sense to couple of programmers in our thirty-ish member team.
> > > > >
> > > > > Virtual memory is by no means necessary to solve this problem easily. We have build
> > > > > servers mostly configured with static page files, so as long as it is possible to
> > > > > just add equivalent amount of RAM, the OOM problem will be solved just as well.
> > > >
> > > > Maybe the only thing you're thinking about is your build system,
> > > > and the only people that care about your build system is you?
> > > >
> > > > Consider a HTTP server - do you want to (e.g.) drop one connection or abort
> > > > one request, or do you want to get killed and drop all current connections?
> > > >
> > > > Consider a word processor - do you want to (e.g.) free memory from the "undo buffer" and retry
> > > > (and if that doesn't work display a dialog box informing the user and save the current document
> > > > and shut down gracefully); or do you want to get killed and lose all the user's unsaved work;
> > > > or do you want kill X11 and screw up every single app that's currently running?
> > > >
> > > > How about a database management engine with 8 GiB of cached data it can easily discard - would you
> > > > want everything that depends on it to suffer "sudden database un-availability" for no reason?
> > >
> > > In my mind, it is "do I want those developers to fix some other bugs or spent
> > > their time on already well handled OOM problem?"
> >
> > My perspective is more like; do I trust software written
> > by people that are so incompetent they don't care if
> > it randomly "crashes" (due to OOM) without warning and without
> > any safeguards (and possibly for no sane reason
> > at all)? If the developers are this stupid, how many other things have they completely screwed up/ignored?
> >
> > > Personally, I can't see what
> > > benefits of not having virtual memory could offset the benefits of having it.
> >
> > I'm not sure if there's some kind of misunderstanding here,
> > or where your "not having virtual memory" came from.
> >
> > I'm mostly advocating "virtual memory, without overcommit (and with swap)" and thought you are
> > advocating "virtual memory, with overcommit (and therefore with OOM killer preventing any software
> > from doing anything correctly even when it's entirely possible and very beneficial)".
> >
> > > If we have page file backed virtual memory, the examples you cite above won't happen that way. Instead
> > > you'll see gradual decrease in performance, which may get really bad before total failure (whether that's
> > > OOM or slowdown so bad that it disrupts the service, depends on virtual memory configuration).
> >
> > No (or not necessarily). For example, if there's 2 GiB of
> > RAM and 20 GiB of swap space; and software frequently
> > uses 1 GiB of data and rarely uses 21 GiB of data; then it will not be slow at all. Note that this is very
> > common for desktop users - e.g. multiple applications and/or multiple browser tabs, with large amounts of
> > data, that are left "running" (sometimes for days) where all the data isn't actually accessed.
> >
> > > The HTTP server example is actually really similar to our build server. If we wished, we could implement
> > > things so that only one operation or part of an operation fails due to OOM. But it makes no sense, since
> > > virtual memory solves the problem. And it works for all the operations and parts of them. And for future
> > > operations we (or some console manufacturer) add. And also for other stuff running on that server.
> >
> > Virtual memory does not solve the problems caused by allowing overcommit and then finding that
> > what the OS committed to is actually needed and doesn't exist, and that "OOM killer" has to screw
> > something up without warning and without any chance of avoidance or graceful mitigation.
> >
> > > It's a nice idea that some process which holds large amounts
> > > of memory for caching purposes could release it
> > > when server starts to run out. But that doesn't really work
> > > on general purpose server. What if there are several
> > > independent processes, but they have different idea of what's
> > > the limit of "server starting to run out of memory"?
> > > The one with more relaxed limit won't free any of its cache before the other one has freed its.
> >
> > A process releasing its own memory when (e.g.) it tries to allocate more and "mmap()" returns ENOMEM and
> > retrying (and succeeding); or cancelling whatever it wanted the extra memory for (and not anything else);
> > or providing feadback directly to user/s (e.g. a dialog box so the user immediately knows what happened
> > and doesn't have to go hunting for obscure logs); or giving the process a chance to save important data
> > to disk so that it's not lost; are all important "first steps" to making software suck less.
> >
> > They are not the last steps by any means. For example; the system can be augmented by
> > per process quotas, or cgroups, and/or by some kind of "low memory notifications" sent
> > out by OS (to ask process/es to release some amount of memory if they're able).

> Well, not over-commit is not exactly side-effect free. Think allocating a large sparse matrix
> and only writing a few pages worth of it, without over-commit wouldn't you need to allocate
> all memory up front (for a truly sparse matrix could that not mean not being able to allocate
> it at all?).

Yes; this is why I advocate "virtual memory, without overcommit (with swap)". The idea is that the memory you reserve up front (that doesn't get used) is mostly just swap space that doesn't get used.

> My humble take on this is, this is a trade-off which requires a policy decision
> by the admin/user as there seem to be valid use cases for all possible combinations, no?

In general; disk space is so much cheaper than RAM (by a factor of around 200?) that for everything except mass produced embedded systems and rare exceptions (e.g. diskless systems, compute farms) its stupid to bother with anything other than "no overcommit, with swap space".

Don't forget that the time user/admins spend trying to find and understand relevant information, figuring out what settings they actually want, and then diddling with settings (and probably misconfiguring them and wondering why the OS "doesn't work right") also has a "potentially not insignificant" cost (e.g. a few hours of an admin's time is likely to cost more than 1 TiB of disk space); and there's a plausible argument that (from a "user experience" perspective, especially for normal consumers) needing to configure something (rather than "works out of the box") is a design flaw.

- Brendan