mpx (mpx@nomail.pl) on 5/29/11 wrote:
>
>Why are you mixing the topics of CPU pages, and caching files?

Because the caches aren't just some kernel internal
thing: they are also exposed directly to user space
through mmap.

Now, you can actually consider mmap() to be a separate
issue from caching - and Linux used to do it long ago.
We had a purely-internal "buffer cache" that was used for
disk caching, and then if somebody mmap'ed a file we
would allocate a page for the mmap.

The problem with that is that it's rather complicated,
and very fragile. Especially in the presence of shared
writable mappings (ie user-land actually writing to the
file through a memory mapping), having separate caches
and mappings is really a nasty piece of business.

Fundamental issue #1: cache coherency when you have two
different models and allocations. Not just the CPU caches
for the actual memory, but the coherency at a software
level when one process does a "write()" system call (or
"truncate()" - that really ends up being interesting),
while another one is accessing the thing just by reading
and writing to a mmap'ed area.

Fundamental issue #2: memory use. You actually want to
try to share the backing store memory, because it turns
out that mmap'ed file-backed memory is a big part of your
memory pressure under some loads - big executables. And
you want to share it naturally and easily, not just to
save memory, but because you want to build up and tear
down processes quickly - so you don't want to copy the
executable image around etc.

And don't get me wrong: "nasty piece of business" is not
the same thing as "cannot be done". I'm 100% sure that it
can be done. Can it be done well? I've never seen
it, and in Linux the solution was certainly "don't do
that then", and we unified the caching in the "page cache"
that does everything at a page granularity, rather than
the filesystem block granularity or whatever.

In other words, I've done it both ways, and I can attest
to the fact that personally I would never ever go
back to the bad old days when caching was somehow a
separate issue from the memory mapping support.

So you don't *have* to mix up the MMU page size with the
filesystem cache, but anybody who doesn't is either
incompetent, doesn't know what they are doing, or is
working on some OS that is irrelevant and specialized
and just never needs to worry too much about small
details like mmap.

Linus