Brendan (btrotter.delete@this.gmail.com) on January 12, 2020 9:19 pm wrote:
> anon (anon.delete@this.b.c) on January 12, 2020 11:30 am wrote:
> >
> > Even if you write your own code extremely carefully, unless you a programming in an embedded context
> > where you wrote all the code running on the platform, you can be pretty damn certain there is a
> > lot of other parts in your system that are not able to gracefully handle OOM. There are just no
> > server or desktop platforms in widespread use that can deal with OOM. It's entirely possible that
> > your program sucks up all the memory, and then OOM is triggered by some crucial background service
> > that nothing can live without, and which does not have viable alternatives to allocating.
> >
> > It's because of this that the sane option for desktop or server is to just ignore OOM and
> > pretend it doesn't exist. Just hardening your own software is the worst kind of idiocy
> > -- it's a massive waste of time, and it doesn't actually protect you from anything.
>
> This kind of thinking is the reason why the world sucks ("Some software is shit, therefore all software
> should be forced to be shit forever and nobody should ever try to make software better").

This assumes there's an easy way to "make software better". Sure, if during large parallel compile system runs out of memory, the compile can be restarted with less parallelism. And when our custom game asset builder runs out of memory, just restart from some previous known good state. Oh wait, that already happens when we just hit "Build" button again (or someone makes an SVN commit and build system automatically start making a new build).

OOMs are practically never a problem for us. Running with not-quite-enough-physical-memory is sometimes a problem, but gets handled nicely by virtual memory. On average, we have plenty of RAM. Sometimes various independent processes with large memory footprint just happen to run at the same time.

If I had to come up with some manual system to handle the problem, I don't really see what I could do to improve it. There could be some heuristics about available memory that would affect how much parallel processes are launched. The problem is coming up with good heuristics. I have no idea how much the shader compiler will need this time, or if it needs to run at all. There's hundreds of variables to consider, each only making sense to couple of programmers in our thirty-ish member team.

Virtual memory is by no means necessary to solve this problem easily. We have build servers mostly configured with static page files, so as long as it is possible to just add equivalent amount of RAM, the OOM problem will be solved just as well.

-JLarja