Tzvetan Mikov (tzvetanmi@yahoo.com) on 5/13/07 wrote:
---------------------------
>Off the top of my head, conceptually, I can't really tell why Windows would have
>trouble with anything up to 2-3GB RAM.

Out of the box, the maximum virtual address space available to a process is 2GB. (The upper 2GB is reserved for the kernel.)

An application that is using much more than 1GB is going to start getting virtual address space fragmentation problems -- if you have a pointer that is pointing to an address at 1.5GB, you're currently using 1GB, and you try to allocate anything over 0.5GB, it will not be able to satisfy that request because the OS can't renumber your pointer. On a 2GB PC you might still have nearly 1GB of physical RAM free, and nearly 1GB of virtual address space free, but you can't allocate 0.5GB.

One of the key features of virtual memory is that you can remap physical addresses under the covers to get rid of memory fragmentation; what do you do when the virtual addresses are fragmented? You need virtual virtual addresses. :-)

A simple example where a nice, high-performance strategy comes unstuck when virtual addresses start to run out is the STL vector's growth behaviour. (It's a dynamically-extendable array, for those who don't know.) To satisfy the C++ Standard's requirement that appending to the end of the vector must be an amortised constant time operation, it grows by a constant percentage of its existing size each time it runs out of space. Under Visual C++ 6, developed at a time when virtual memory was plentiful, it would double each time; under VS2005 it grows by 50% each time. The latter runs out of memory less often at the expense of copying data more frequently. Doubling the size shouldn't actually cause physical memory to be wasted, because the pages will only be mapped when they are touched; but when virtual memory is scarce the append operation fails depressingly often, even when there is plenty of RAM available.

>So it is conceivable that the trouble you app is having with more than 1.5GB RAM
>under XP may not be so much a HIGHMEM issue, but a matter of XP not being optimized
>for such amount of RAM. The kernel is pretty old. So, it would be interesting to run the same test with 32-bit Vista.

It's not the kernel having the problem, it's the application not being able to use the full 2GB of physical (and virtual) address space that should be available to it, and one of the big causes of that is virtual address space fragmentation.

The point is that it starts to be an issue way before the theoretical 4GB limitation of 32 bit computers. Anything that attempts to use much more than half of a 2GB system's memory needs to take care to avoid having problems, and a memory footprint of "more than 1GB" is a lot less unusual for a modern desktop app than "4GB".