Pre-populating anonymous pages

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), June 10, 2019 11:20 am
Room: Moderated Discussions
Brendan (btrotter.delete@this.gmail.com) on June 9, 2019 3:29 am wrote:
>
> I assumed (incorrectly) that it's not too horrible, Travis points out that it actually is a little
> horrible, then you point out that it's worse than Travis said because "I don't intend to use this area
> at all (I'm planning to carve it up)" is conflated with "I might use this memory but not soon"?

No, the fundamental issue is that it's not actually horrible at all.

I realize that when people talk performance, they always talk throughput. That's bullshit, but it's easy to explain: it's generally the number that is easy to generate and point at. There are a lot of throughput benchmarks, they are easy, and they are often almost completely irrelevant.

Because throughput is not a primary performance measure, and it's generally not even what you should think of when it comes to mmap. Latency is much more important. Arguably the only time throughput is actually important is when it becomes "latency at a big level".

But latency is a lot harder to actually measure and talk about, except at that large "whole job" level. It's really annoying and bad when it happens to be noticeable (and depending on what "notices", we may be talking about microseconds or milliseconds or bigger timeframes), but it's hard to get your tools to do latency at almost all levels.

And the latency of pre-populating everything is often absolutely horrendous.

The number of people who actually care about pre-populating mmap is very very low, and those people actually usually don't do it for performance in the first place, and anyway they already have the interfaces to do so (either with MAP_PROTECT, or with mlock/madvise).

So the point I was trying to make is that pre-populating is usually the wrong thing to do. I gave a couple of fundamental reasons, but I guess I didn't articulate well just how fundamental the whole "pre-populating is usually bad" is.

You're hung up about the wrong kind of performance, on the wrong kind of load. For no good reason.

In real life, a lot of mmap's are randomly accessed, and somewhat sparsely so. Pre-populating it is simply fundamentally wrong.

For the most common case (notably executables) we instead do fault-around, which does actually help, and has almost none of the downsides of pre-populating. Fault-around doesn't have any real latency issues, and it also doesn't have any issues with the example I gave about carving up the memory map. Once you start accessing the memory, you'd better be done setting it up, but equally importantly it's just taking advantage of the usual expectations of locality etc. You can turn the fault-around off by explicitly saying you're doing random access, but once again the default is to do the thing that is good for most cases.

So stop looking at irrelevant throughput numbers that make no sense.

That's why I told Travis that if he has a real load that shows this, then we might extend our fault-around to anonymous pages too. Again, note: fault-around, not pre-populate. Doing fault-around is in some sense a bit more complex, but it really is a much better model for mmap.

Basically:

(a) if you want to optimize for throughput and you know exactly what your access patterns are, you already have the tools to do so.

Note that MAP_POPULATE and mlock etc are actually seldom (effectively never) about performance. They tend to be very much about real-time or security issues, where you want to make sure things are in memory not because it's faster, but because you don't want timing jitter, or you don't want things to leak to disk due to paging.

(b) the whole "touch one byte per page once" model is insane. Yes, it hurts when you do that if you don't pre-populate, because you will take a page fault for each access. But nobody sane does that, so the system should damn well not optimize for stupidity.

(c) "performance" is not just about throughput (much less throughput on insane and pointless loads), and on-demand behavior has absolutely huge latency and performance advantages. And yes, the "on-demand" could be smarter, but seriously, nobody has really complained. We only do that smarter thing (the fault-around) for file mappings, because that's where the big wins have been.

Basically, pre-populating things really does hurt you on many real cases, and has no discernible win. But if you know you want to do it, you can do it.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Pre-populating anonymous pagesTravis Downs2019/06/05 04:48 PM
  Pre-populating anonymous pagesJeff S.2019/06/05 08:03 PM
    Pre-populating anonymous pagesTravis Downs2019/06/06 07:11 AM
      Pre-populating anonymous pagesJeff S.2019/06/06 08:40 AM
        Pre-populating anonymous pagesTravis Downs2019/06/06 08:59 AM
          Pre-populating anonymous pagesJeff S.2019/06/06 09:19 AM
  Pre-populating anonymous pagesFoo_2019/06/06 12:30 AM
    Pre-populating anonymous pagesTravis Downs2019/06/06 06:59 AM
      Pre-populating anonymous pagesFoo_2019/06/06 07:56 AM
        Pre-populating anonymous pagesTravis Downs2019/06/06 09:02 AM
  Pre-populating anonymous pagesLinus Torvalds2019/06/06 11:01 AM
    Pre-populating anonymous pagesTravis Downs2019/06/07 02:16 PM
      Pre-populating anonymous pagesBrendan2019/06/08 02:55 AM
        Pre-populating anonymous pagesTravis Downs2019/06/08 08:18 AM
        Pre-populating anonymous pagesLinus Torvalds2019/06/08 11:43 AM
          Pre-populating anonymous pagesBrendan2019/06/09 03:29 AM
            Pre-populating anonymous pagesLinus Torvalds2019/06/10 11:20 AM
          Pre-populating anonymous pagesTravis Downs2019/06/17 09:18 AM
            Pre-populating anonymous pagesLinus Torvalds2019/06/18 04:28 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell avocado?