Tzvetan Mikov (tzvetanmi@yahoo.com) on 5/13/07 wrote:
>
>Can you please elaborate on the practical 1 GB limit a
>little further ?

Since the i386 has a 32-bit total limit on virtual
address space (the segmentation doesn't extend that, it
all works within the 4GB limit), and since user space needs
to be mapped in that area too, you effectively have the
requirement that

- user space virtual memory + kernel virtual memory = 4GB

Some other architectures effectively have separate virtual
memory spaces for user and kernel space, so they don't
necessarily need to be 4GB combined, they can be separately
4GB each. On x86, that isn't practical (you can emulate it
by switching page tables around a lot, and yes, it's been
done, but despite the fact that Intel makes TLB lookups
very fast, it's quite nasty for performance!)

And the thing is, user space really does want its own
virtual memory too. Under Linux, you can choose how to
split up the 4GB address space, but on the whole, you'd
actually prefer to give user space as much as possible,
since

- user space tends to use reasonably sparse virtual
address spaces (loadable libraries in one place, code,
data, stack in other areas etc)

- quite often, you have one "major" app, which wants to
use as much of the physical memory as possible, eg a
database, so you actually want user space to map as much
of the physical memory too!

So Linux defaults to giving user space 3GB, and just 1GB
for the kernel. That's basically the "user space is more
important in the end" version. I think WinXP does a 2:2GB
split.

So basically, the kernel really never has 4GB of virtual
memory. It has 1GB (possibly 2GB).

It also needs to map in PCI memory-mapped stuff etc, so in
practice, it's actually less than that. And then the kernel
will need some more virtual memory for its own use, eg
things like loading modules, and just for the window to do
physical addresses bigger than it can map directly.

So in practice, in Linux, when you had more than about
920MB (I forget the exact number), you ended up maxing out
what the kernel could map directly.

And once you max that out, think about what happens for
a moment...

Once you start having to access physical memory through
some kind of HIGHMEM.SYS window (which is what PAE is),
and cannot map it all, you can no longer keep normal
pointers to such memory around. Instead, you are basically
using a really ugly and strange segmented architecture,
where you keep some kind of indirect pointer, and every
time before you use it, you have to map it into the virtual
address space, access it, and then unmap it again.

So performance plummets, and the code actually gets really
nasty too, so you don't actually use the high memory for
any random data, you only use it for special stuff. As
an example, you'd use it for disk caching (that's what 90%
of all HIGHMEM.SYS usage was too - a lot of people just
set it all - or at least a big chunk of it - aside as a
harddisk cache, because so few programs could use it very
well for anything else).

Linux used it for more than disk caching, but the point
is, since it's special memory that you have to access
through a small window, or can just map into user space,
it's not generally usable any more.

And yes, there were serious problems. In theory, you
can have 64GB of RAM with Linux on a PAE x86 box. In
practice, it seldom worked very well past the 4GB mark,
so PAE itself was almost totally useless. The reason was
that once you had more than 4GB of memory, you usually
had filled out a large chunk of the easily accessible
memory with just all the data structures to keep track
of the rest of memory (that's exaggerated, but it's not
entirely off).

So you literally were in this situation where you had
tons of memory that could be used for some things,
but you ran out of memory for other basic bookkeeping
stuff.

So some people used 16GB of RAM, and by limiting their
working set to cases where the HIGHMEM stuff was fine,
and they didn't need a whole lot of other memory, they
made it work. But the code was (and is) grotty and pretty
disgusting, and performs much worse than just allowing
anything to use any of the RAM.

And the limit literally isn't the 4GB mark.

>So, in practical terms, is there benefit from using
>64-bit OS on a machine with, lets say, 2-3 GB RAM ?

Absolutely. No question what-so-ever. If you have 2GB
of RAM on a 64-bit architecture, you can access it all
easily in the kernel, and none of it is limited to just
some special use. And a single big process can also use
it effectively at the same time, so a database can
actually have it all mapped without having to play
windowing games in user space either.

Linus