Hi,

Jukka Larja (roskakori2006.delete@this.gmail.com) on January 14, 2020 6:37 am wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 12:02 pm wrote:
> > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 8:43 am wrote:
> > > Brendan (btrotter.delete@this.gmail.com) on January 13, 2020 7:21 am wrote:
> > > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 13, 2020 6:41 am wrote:
> > > > > Brendan (btrotter.delete@this.gmail.com) on January 12, 2020 9:19 pm wrote:
> > > > > > anon (anon.delete@this.b.c) on January 12, 2020 11:30 am wrote:
> > > > > > >
> > > > > > > Even if you write your own code extremely carefully, unless you a programming in an embedded context
> > > > > > > where you wrote all the code running on the platform, you can be pretty damn certain there is a
> > > > > > > lot of other parts in your system that are not able to gracefully handle OOM. There are just no
> > > > > > > server or desktop platforms in widespread use that can deal with OOM. It's entirely possible that
> > > > > > > your program sucks up all the memory, and then OOM is triggered by some crucial background service
> > > > > > > that nothing can live without, and which does not have viable alternatives to allocating.
> > > > > > >
> > > > > > > It's because of this that the sane option for desktop or server is to just ignore OOM and
> > > > > > > pretend it doesn't exist. Just hardening your own software is the worst kind of idiocy
> > > > > > > -- it's a massive waste of time, and it doesn't actually protect you from anything.
> > > > > >
> > > > > > This kind of thinking is the reason why the world sucks ("Some software is shit, therefore all software
> > > > > > should be forced to be shit forever and nobody should ever try to make software better").
> > > > >
> > > > > This assumes there's an easy way to "make software better". Sure, if during large parallel
> > > > > compile system runs out of memory, the compile can be restarted with less parallelism. And
> > > > > when our custom game asset builder runs out of memory, just restart from some previous known
> > > > > good state. Oh wait, that already happens when we just hit "Build" button again (or someone
> > > > > makes an SVN commit and build system automatically start making a new build).
> > > > >
> > > > > OOMs are practically never a problem for us. Running with not-quite-enough-physical-memory is sometimes
> > > > > a problem, but gets handled nicely by virtual memory. On average, we have plenty of RAM. Sometimes
> > > > > various independent processes with large memory footprint just happen to run at the same time.
> > > > >
> > > > > If I had to come up with some manual system to handle the problem, I don't really see what I could
> > > > > do to improve it. There could be some heuristics about available memory that would affect how much
> > > > > parallel processes are launched. The problem is coming up with good heuristics. I have no idea how
> > > > > much the shader compiler will need this time, or if it needs to run at all. There's hundreds of variables
> > > > > to consider, each only making sense to couple of programmers in our thirty-ish member team.
> > > > >
> > > > > Virtual memory is by no means necessary to solve this problem easily. We have build
> > > > > servers mostly configured with static page files, so as long as it is possible to
> > > > > just add equivalent amount of RAM, the OOM problem will be solved just as well.
> > > >
> > > > Maybe the only thing you're thinking about is your build system,
> > > > and the only people that care about your build system is you?
> > > >
> > > > Consider a HTTP server - do you want to (e.g.) drop one connection or abort
> > > > one request, or do you want to get killed and drop all current connections?
> > > >
> > > > Consider a word processor - do you want to (e.g.) free memory from the "undo buffer" and retry
> > > > (and if that doesn't work display a dialog box informing the user and save the current document
> > > > and shut down gracefully); or do you want to get killed and lose all the user's unsaved work;
> > > > or do you want kill X11 and screw up every single app that's currently running?
> > > >
> > > > How about a database management engine with 8 GiB of cached data it can easily discard - would you
> > > > want everything that depends on it to suffer "sudden database un-availability" for no reason?
> > >
> > > In my mind, it is "do I want those developers to fix some other bugs or spent
> > > their time on already well handled OOM problem?"
> >
> > My perspective is more like; do I trust software written
> > by people that are so incompetent they don't care if
> > it randomly "crashes" (due to OOM) without warning and without
> > any safeguards (and possibly for no sane reason
> > at all)? If the developers are this stupid, how many other things have they completely screwed up/ignored?
>
> There's always compromizes to make. Since OOM practically never happens during normal running (and would
> equally seldom be avoidable with any sensible tricks), My view of preparing for it is the same as with any
> (premature) micro-optimization. It doesn't hurt, but I wouldn't want to pay people much for doing it.

Some things I'm just unwilling to accept. For example, I can accept software failing due to OOM, but I can't accept software not failing gracefully (losing data, not informing user of the problem in the same way it provides feedback of other error conditions, etc).

> I'm very familiar with OOM on consoles (where there is no virtual memory), and we have thought
> about possibly doing some last-resortish in that case (Unreal Engine, for example, has by default
> 32 MB extra OOM buffer for tight spots). That, however, is on a platform where one has a promise
> of certain amount of memory. We haven't actually done anything, because doing it right is hard
> (consider that memory manager may run out of memory while several threads are requesting more.
> Freeing some cache or other unimportant data in such environment is asking for deadlocks and random
> crashes) and benefits compared to just reducing overall memory usage would be too small.
>
> > > Personally, I can't see what
> > > benefits of not having virtual memory could offset the benefits of having it.
> >
> > I'm not sure if there's some kind of misunderstanding here,
> > or where your "not having virtual memory" came from.
>
> Sorry, I must have mixed up your argument with someone else's at some point.
>
> > I'm mostly advocating "virtual memory, without overcommit (and with swap)" and thought you are
> > advocating "virtual memory, with overcommit (and therefore with OOM killer preventing any software
> > from doing anything correctly even when it's entirely possible and very beneficial)".
>
> I mostly don't care, since I just increase page file size until OOMs don't happen anymore. I think
> Windows doesn't overcommit and that's the only platform I'm really familiar with (from those platforms
> that have virtual memory). OOM induced crash on our build servers basically means we should reboot
> anyway, because it's too much trouble to check if something important has crashed.

Yeah, Windows does a lot of things right, and I suspect that at least part of the problem on *nix operating systems is "fork()". Specifically; if you have a large process with many GiB of data, "fork()" means the OS would have to commit to being able to provide copies of most of that data (anything writable becomes "potentially copy on writable"), even though it's very unlikely to be needed (it's likely the child won't modify much before it does "exec()" anyway), and this makes overcommit a lot more attractive.

> > > If we have page file backed virtual memory, the examples you cite above won't happen that way. Instead
> > > you'll see gradual decrease in performance, which may get really bad before total failure (whether that's
> > > OOM or slowdown so bad that it disrupts the service, depends on virtual memory configuration).
> >
> > No (or not necessarily). For example, if there's 2 GiB of
> > RAM and 20 GiB of swap space; and software frequently
> > uses 1 GiB of data and rarely uses 21 GiB of data; then it will not be slow at all. Note that this is very
> > common for desktop users - e.g. multiple applications and/or multiple browser tabs, with large amounts of
> > data, that are left "running" (sometimes for days) where all the data isn't actually accessed.
>
> I have hard time seeing that as anything else but memory leak. In case of memory leak it doesn't
> matter what you do. You'll run out of any amount of physical and virtual memory, with or without
> overcommit (except that with overcommit one can leak huge amounts of untouched memory).

I have a hard time seeing how anyone can consider it a memory leak. If the user opens 10 applications and leaves 9 of them "running/idle" while they use one of them, then none of them are leaking any memory (but 9 applications that are "running/idle" aren't actually using any of the memory).

You said you work on game consoles. If I'm playing a game that consumes 6 GiB of memory, but pause the game for a few hours while I go shopping, has that 6 GiB of memory been leaked? Does the memory become "unleaked" again when I unpause and continue playing the game later?

> > > It's a nice idea that some process which holds large amounts
> > > of memory for caching purposes could release it
> > > when server starts to run out. But that doesn't really work
> > > on general purpose server. What if there are several
> > > independent processes, but they have different idea of what's
> > > the limit of "server starting to run out of memory"?
> > > The one with more relaxed limit won't free any of its cache before the other one has freed its.
> >
> > A process releasing its own memory when (e.g.) it tries to allocate more and "mmap()" returns ENOMEM and
> > retrying (and succeeding); or cancelling whatever it wanted the extra memory for (and not anything else);
> > or providing feadback directly to user/s (e.g. a dialog box so the user immediately knows what happened
> > and doesn't have to go hunting for obscure logs); or giving the process a chance to save important data
> > to disk so that it's not lost; are all important "first steps" to making software suck less.
>
> You can't presume any of those actions will succeed, if the original memory allocation failed,
> unless you somehow know that the OS can do everything you ask without using any extra memory itself.
> Even if you free significant amount yourself, another process may be hogging it immediately. Actually,
> unless you are very careful, another thread in your own process may be hogging it.
>
> So basically you are doing complicated things (that will hardly ever get tested
> and will most likely be buggy anyway) and still can't guarantee a good outcome.

You've implied that you're a software developer, and yet you think this is "complicated" and "untestable"?

> > They are not the last steps by any means. For example; the system can be augmented by
> > per process quotas, or cgroups, and/or by some kind of "low memory notifications" sent
> > out by OS (to ask process/es to release some amount of memory if they're able).
>
> I'm pretty sure all OSes offer APIs to ask for amount of available memory. I don't believe
> it matters that it's "pull" instead of "push".

Having to constantly monitor the OS (rather than the OS notifying you) would increase the hassle; especially for "otherwise idle" processes.

> Most projects have better things to do.

If most people were incompetent, would you make incompetence mandatory?

- Brendan