Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 12:02 pm wrote:
> Hi,
>
> Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 8:43 am wrote:
> > Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 7:21 am wrote:
> > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 6:41 am wrote:
> > > > Brendan (btrotter.delete@this.gmail.com) on January 12, 2020 9:19 pm wrote:
> > > > > anon (anon.delete@this.b.c) on January 12, 2020 11:30 am wrote:
> > > > > >
> > > > > > Even if you write your own code extremely carefully, unless you a programming in an embedded context
> > > > > > where you wrote all the code running on the platform, you can be pretty damn certain there is a
> > > > > > lot of other parts in your system that are not able to gracefully handle OOM. There are just no
> > > > > > server or desktop platforms in widespread use that can deal with OOM. It's entirely possible that
> > > > > > your program sucks up all the memory, and then OOM is triggered by some crucial background service
> > > > > > that nothing can live without, and which does not have viable alternatives to allocating.
> > > > > >
> > > > > > It's because of this that the sane option for desktop or server is to just ignore OOM and
> > > > > > pretend it doesn't exist. Just hardening your own software is the worst kind of idiocy
> > > > > > -- it's a massive waste of time, and it doesn't actually protect you from anything.
> > > > >
> > > > > This kind of thinking is the reason why the world sucks ("Some software is shit, therefore all software
> > > > > should be forced to be shit forever and nobody should ever try to make software better").
> > > >
> > > > This assumes there's an easy way to "make software better". Sure, if during large parallel
> > > > compile system runs out of memory, the compile can be restarted with less parallelism. And
> > > > when our custom game asset builder runs out of memory, just restart from some previous known
> > > > good state. Oh wait, that already happens when we just hit "Build" button again (or someone
> > > > makes an SVN commit and build system automatically start making a new build).
> > > >
> > > > OOMs are practically never a problem for us. Running with not-quite-enough-physical-memory is sometimes
> > > > a problem, but gets handled nicely by virtual memory. On average, we have plenty of RAM. Sometimes
> > > > various independent processes with large memory footprint just happen to run at the same time.
> > > >
> > > > If I had to come up with some manual system to handle the problem, I don't really see what I could
> > > > do to improve it. There could be some heuristics about available memory that would affect how much
> > > > parallel processes are launched. The problem is coming up with good heuristics. I have no idea how
> > > > much the shader compiler will need this time, or if it needs to run at all. There's hundreds of variables
> > > > to consider, each only making sense to couple of programmers in our thirty-ish member team.
> > > >
> > > > Virtual memory is by no means necessary to solve this problem easily. We have build
> > > > servers mostly configured with static page files, so as long as it is possible to
> > > > just add equivalent amount of RAM, the OOM problem will be solved just as well.
> > >
> > > Maybe the only thing you're thinking about is your build system,
> > > and the only people that care about your build system is you?
> > >
> > > Consider a HTTP server - do you want to (e.g.) drop one connection or abort
> > > one request, or do you want to get killed and drop all current connections?
> > >
> > > Consider a word processor - do you want to (e.g.) free memory from the "undo buffer" and retry
> > > (and if that doesn't work display a dialog box informing the user and save the current document
> > > and shut down gracefully); or do you want to get killed and lose all the user's unsaved work;
> > > or do you want kill X11 and screw up every single app that's currently running?
> > >
> > > How about a database management engine with 8 GiB of cached data it can easily discard - would you
> > > want everything that depends on it to suffer "sudden database un-availability" for no reason?
> >
> > In my mind, it is "do I want those developers to fix some other bugs or spent
> > their time on already well handled OOM problem?"
>
> My perspective is more like; do I trust software written by people that are so incompetent they don't care if
> it randomly "crashes" (due to OOM) without warning and without any safeguards (and possibly for no sane reason
> at all)? If the developers are this stupid, how many other things have they completely screwed up/ignored?
>
> > Personally, I can't see what
> > benefits of not having virtual memory could offset the benefits of having it.
>
> I'm not sure if there's some kind of misunderstanding here,
> or where your "not having virtual memory" came from.
>
> I'm mostly advocating "virtual memory, without overcommit (and with swap)" and thought you are
> advocating "virtual memory, with overcommit (and therefore with OOM killer preventing any software
> from doing anything correctly even when it's entirely possible and very beneficial)".
>
> > If we have page file backed virtual memory, the examples you cite above won't happen that way. Instead
> > you'll see gradual decrease in performance, which may get really bad before total failure (whether that's
> > OOM or slowdown so bad that it disrupts the service, depends on virtual memory configuration).
>
> No (or not necessarily). For example, if there's 2 GiB of RAM and 20 GiB of swap space; and software frequently
> uses 1 GiB of data and rarely uses 21 GiB of data; then it will not be slow at all. Note that this is very
> common for desktop users - e.g. multiple applications and/or multiple browser tabs, with large amounts of
> data, that are left "running" (sometimes for days) where all the data isn't actually accessed.
>
> > The HTTP server example is actually really similar to our build server. If we wished, we could implement
> > things so that only one operation or part of an operation fails due to OOM. But it makes no sense, since
> > virtual memory solves the problem. And it works for all the operations and parts of them. And for future
> > operations we (or some console manufacturer) add. And also for other stuff running on that server.
>
> Virtual memory does not solve the problems caused by allowing overcommit and then finding that
> what the OS committed to is actually needed and doesn't exist, and that "OOM killer" has to screw
> something up without warning and without any chance of avoidance or graceful mitigation.
>
> > It's a nice idea that some process which holds large amounts
> > of memory for caching purposes could release it
> > when server starts to run out. But that doesn't really work
> > on general purpose server. What if there are several
> > independent processes, but they have different idea of what's
> > the limit of "server starting to run out of memory"?
> > The one with more relaxed limit won't free any of its cache before the other one has freed its.
>
> A process releasing its own memory when (e.g.) it tries to allocate more and "mmap()" returns ENOMEM and
> retrying (and succeeding); or cancelling whatever it wanted the extra memory for (and not anything else);
> or providing feadback directly to user/s (e.g. a dialog box so the user immediately knows what happened
> and doesn't have to go hunting for obscure logs); or giving the process a chance to save important data
> to disk so that it's not lost; are all important "first steps" to making software suck less.
>
> They are not the last steps by any means. For example; the system can be augmented by
> per process quotas, or cgroups, and/or by some kind of "low memory notifications" sent
> out by OS (to ask process/es to release some amount of memory if they're able).
>
> - Brendan
>

Well, not over-commit is not exactly side-effect free. Think allocating a large sparse matrix and only writing a few pages worth of it, without over-commit wouldn't you need to allocate all memory up front (for a truly sparse matrix could that not mean not being able to allocate it at all?). My humble take on this is, this is a trade-off which requires a policy decision by the admin/user as there seem to be valid use cases for all possible combinations, no?