By: JasonB (no.delete@this.spam.com), May 14, 2007 12:50 am
Room: Moderated Discussions
Tzvetan Mikov (tzvetanmi@yahoo.com) on 5/13/07 wrote:
---------------------------
>Off the top of my head, conceptually, I can't really tell why Windows would have
>trouble with anything up to 2-3GB RAM.
Out of the box, the maximum virtual address space available to a process is 2GB. (The upper 2GB is reserved for the kernel.)
An application that is using much more than 1GB is going to start getting virtual address space fragmentation problems -- if you have a pointer that is pointing to an address at 1.5GB, you're currently using 1GB, and you try to allocate anything over 0.5GB, it will not be able to satisfy that request because the OS can't renumber your pointer. On a 2GB PC you might still have nearly 1GB of physical RAM free, and nearly 1GB of virtual address space free, but you can't allocate 0.5GB.
One of the key features of virtual memory is that you can remap physical addresses under the covers to get rid of memory fragmentation; what do you do when the virtual addresses are fragmented? You need virtual virtual addresses. :-)
A simple example where a nice, high-performance strategy comes unstuck when virtual addresses start to run out is the STL vector's growth behaviour. (It's a dynamically-extendable array, for those who don't know.) To satisfy the C++ Standard's requirement that appending to the end of the vector must be an amortised constant time operation, it grows by a constant percentage of its existing size each time it runs out of space. Under Visual C++ 6, developed at a time when virtual memory was plentiful, it would double each time; under VS2005 it grows by 50% each time. The latter runs out of memory less often at the expense of copying data more frequently. Doubling the size shouldn't actually cause physical memory to be wasted, because the pages will only be mapped when they are touched; but when virtual memory is scarce the append operation fails depressingly often, even when there is plenty of RAM available.
>So it is conceivable that the trouble you app is having with more than 1.5GB RAM
>under XP may not be so much a HIGHMEM issue, but a matter of XP not being optimized
>for such amount of RAM. The kernel is pretty old. So, it would be interesting to run the same test with 32-bit Vista.
It's not the kernel having the problem, it's the application not being able to use the full 2GB of physical (and virtual) address space that should be available to it, and one of the big causes of that is virtual address space fragmentation.
The point is that it starts to be an issue way before the theoretical 4GB limitation of 32 bit computers. Anything that attempts to use much more than half of a 2GB system's memory needs to take care to avoid having problems, and a memory footprint of "more than 1GB" is a lot less unusual for a modern desktop app than "4GB".
---------------------------
>Off the top of my head, conceptually, I can't really tell why Windows would have
>trouble with anything up to 2-3GB RAM.
Out of the box, the maximum virtual address space available to a process is 2GB. (The upper 2GB is reserved for the kernel.)
An application that is using much more than 1GB is going to start getting virtual address space fragmentation problems -- if you have a pointer that is pointing to an address at 1.5GB, you're currently using 1GB, and you try to allocate anything over 0.5GB, it will not be able to satisfy that request because the OS can't renumber your pointer. On a 2GB PC you might still have nearly 1GB of physical RAM free, and nearly 1GB of virtual address space free, but you can't allocate 0.5GB.
One of the key features of virtual memory is that you can remap physical addresses under the covers to get rid of memory fragmentation; what do you do when the virtual addresses are fragmented? You need virtual virtual addresses. :-)
A simple example where a nice, high-performance strategy comes unstuck when virtual addresses start to run out is the STL vector's growth behaviour. (It's a dynamically-extendable array, for those who don't know.) To satisfy the C++ Standard's requirement that appending to the end of the vector must be an amortised constant time operation, it grows by a constant percentage of its existing size each time it runs out of space. Under Visual C++ 6, developed at a time when virtual memory was plentiful, it would double each time; under VS2005 it grows by 50% each time. The latter runs out of memory less often at the expense of copying data more frequently. Doubling the size shouldn't actually cause physical memory to be wasted, because the pages will only be mapped when they are touched; but when virtual memory is scarce the append operation fails depressingly often, even when there is plenty of RAM available.
>So it is conceivable that the trouble you app is having with more than 1.5GB RAM
>under XP may not be so much a HIGHMEM issue, but a matter of XP not being optimized
>for such amount of RAM. The kernel is pretty old. So, it would be interesting to run the same test with 32-bit Vista.
It's not the kernel having the problem, it's the application not being able to use the full 2GB of physical (and virtual) address space that should be available to it, and one of the big causes of that is virtual address space fragmentation.
The point is that it starts to be an issue way before the theoretical 4GB limitation of 32 bit computers. Anything that attempts to use much more than half of a 2GB system's memory needs to take care to avoid having problems, and a memory footprint of "more than 1GB" is a lot less unusual for a modern desktop app than "4GB".
Topic | Posted By | Date |
---|---|---|
Rock/Tukwila rumors | mas | 2007/05/05 11:59 AM |
Rock/Tukwila rumors | David Kanter | 2007/05/05 01:33 PM |
Rock/Tukwila rumors | Dean Kent | 2007/05/05 02:35 PM |
K8 vs Win64 timeline | anonymous | 2007/05/05 05:19 PM |
Yes, I misremembered... | Dean Kent | 2007/05/05 09:03 PM |
Rock | Daniel Bizó | 2007/05/06 01:34 AM |
Rock | Dean Kent | 2007/05/06 06:11 AM |
Rock/Tukwila rumors | Joe | 2007/05/06 10:24 AM |
Rock/Tukwila rumors | Dean Kent | 2007/05/06 10:49 AM |
Rock/Tukwila rumors | Linus Torvalds | 2007/05/06 11:09 AM |
Rock/Tukwila rumors | anon | 2007/05/07 12:32 AM |
Rock/Tukwila rumors | Rakesh Malik | 2007/05/07 08:36 AM |
Rock/Tukwila rumors | Michael S | 2007/05/07 09:06 AM |
Rock/Tukwila rumors | anon | 2007/05/07 08:48 PM |
Rock/Tukwila rumors | Rakesh Malik | 2007/05/08 05:45 AM |
Rock/Tukwila rumors | anon | 2007/05/08 04:30 PM |
Wow. (nt) | Brannon | 2007/05/08 05:16 PM |
Rock/Tukwila rumors | rwessel | 2007/05/08 08:48 PM |
Rock/Tukwila rumors | JS | 2007/05/08 09:07 PM |
Rock/Tukwila rumors | JS | 2007/05/09 05:44 AM |
Rock/Tukwila rumors | Rakesh Malik | 2007/05/09 04:35 AM |
Much ado about x | Michael S | 2007/05/09 08:39 AM |
Call it x86-64 | Linus Torvalds | 2007/05/09 09:27 AM |
(i)AMD64 | Michael S | 2007/05/09 11:16 AM |
(i)AMD64 | Linus Torvalds | 2007/05/09 11:29 AM |
(i)AMD64 | Groo | 2007/05/09 03:45 PM |
TIFNAA | anonymous | 2007/05/09 04:49 PM |
Inspired by FYR Macedonia? (NT) | Michael S | 2007/05/09 10:21 PM |
More likely... | rwessel | 2007/05/09 11:39 PM |
TIFNAA | Gabriele Svelto | 2007/05/09 10:57 PM |
(i)AMD64 | James | 2007/05/10 01:27 AM |
i86 | Dean Kent | 2007/05/09 11:30 AM |
(i)AMD64 | Max | 2007/05/09 12:28 PM |
wide86? long86? | hobold | 2007/05/10 04:05 AM |
x87 perhaps, it is one more. :) (NT) | Groo | 2007/05/10 04:50 AM |
x86+ | Dean Kent | 2007/05/10 07:44 AM |
Does it really matter? | Doug Siebert | 2007/05/10 08:10 AM |
let's stay with x86-64 for now, please | Marcin Niewiadomski | 2007/05/10 10:50 AM |
let's stay with x86-64 for now, please | Dean Kent | 2007/05/11 05:11 AM |
let's stay with x86-64 for now, please | rwessel | 2007/05/11 01:46 PM |
let's stay with x86-64 for now, please | Dean Kent | 2007/05/11 05:03 PM |
let's stay with x86-64 for now, please | Michael S | 2007/05/12 09:49 AM |
let's stay with x86-64 for now, please | Dean Kent | 2007/05/12 12:05 PM |
let's stay with x86-64 for now, please | Michael S | 2007/05/12 12:25 PM |
let's stay with x86-64 for now, please | Dean Kent | 2007/05/12 02:39 PM |
let's stay with x86-64 for now, please | JasonB | 2007/05/13 06:43 AM |
client consolidation | Michael S | 2007/05/13 07:37 AM |
let's stay with x86-64 for now, please | Tzvetan Mikov | 2007/05/13 02:44 PM |
let's stay with x86-64 for now, please | rwessel | 2007/05/14 01:42 PM |
What's your point? | Doug Siebert | 2007/05/11 01:56 PM |
What's your point? | Linus Torvalds | 2007/05/11 03:15 PM |
What's your point? | Doug Siebert | 2007/05/13 02:11 PM |
What's your point? | Dean Kent | 2007/05/13 06:04 PM |
What's your point? | JasonB | 2007/05/14 01:06 AM |
What's your point? | Dean Kent | 2007/05/14 06:20 AM |
What's your point? | JasonB | 2007/05/14 03:35 PM |
What's your point? | JasonB | 2007/05/14 06:35 PM |
What's your point? | Dean Kent | 2007/05/14 07:12 PM |
What's your point? | Dean Kent | 2007/05/11 05:06 PM |
What's your point? | Stephen H | 2007/05/13 12:55 AM |
Why didn't MS take advantage of PAE? | David W. Hess | 2007/05/13 07:37 AM |
PAE sucks (Why didn't MS take advantage of PAE?) | Linus Torvalds | 2007/05/13 09:20 AM |
PAE sucks (Why didn't MS take advantage of PAE?) | Dean Kent | 2007/05/13 09:49 AM |
PAE sucks (Why didn't MS take advantage of PAE?) | David W. Hess | 2007/05/13 11:37 AM |
> 1 GB RAM on a 32-bit system | Tzvetan Mikov | 2007/05/13 12:44 PM |
> 1 GB RAM on a 32-bit system | S. Rao | 2007/05/13 02:00 PM |
> 1 GB RAM on a 32-bit system | Tzvetan Mikov | 2007/05/13 04:32 PM |
> 1 GB RAM on a 32-bit system | S. Rao | 2007/05/13 11:19 PM |
> 1 GB RAM on a 32-bit system | Linus Torvalds | 2007/05/13 02:46 PM |
> 1 GB RAM on a 32-bit system | Tzvetan Mikov | 2007/05/13 04:23 PM |
> 1 GB RAM on a 32-bit system | JasonB | 2007/05/13 05:37 PM |
Windows manages memory differently | Tzvetan Mikov | 2007/05/13 07:31 PM |
Windows manages memory differently | JasonB | 2007/05/14 12:50 AM |
Windows manages memory differently | Tzvetan Mikov | 2007/05/14 07:56 AM |
Windows manages memory differently | rwessel | 2007/05/14 02:40 PM |
Windows manages memory differently | David W. Hess | 2007/05/14 03:07 PM |
Windows manages memory differently | rwessel | 2007/05/14 03:51 PM |
Windows manages memory differently | Tzvetan Mikov | 2007/05/14 04:40 PM |
Windows manages memory differently | rwessel | 2007/05/14 05:09 PM |
Windows manages memory differently | Howard Chu | 2007/05/14 10:17 AM |
Windows manages memory differently | Jukka Larja | 2007/05/14 10:30 AM |
Windows manages memory differently | Tzvetan Mikov | 2007/05/14 12:54 PM |
Windows manages memory differently | Howard Chu | 2007/05/15 02:35 AM |
Windows manages memory differently | Groo | 2007/05/15 06:34 AM |
Anyone know what OS X (10.4, Intel, desktop) does? | Matt Sayler | 2007/05/15 05:23 AM |
Anyone know what OS X (10.4, Intel, desktop) does? | Wes Felter | 2007/05/15 07:37 AM |
Anyone know what OS X (10.4, Intel, desktop) does? | Anonymous | 2007/05/15 09:49 AM |
Anyone know what OS X (10.4, Intel, desktop) does? | anon2 | 2007/05/15 06:13 PM |
PAE sucks (Why didn't MS take advantage of PAE?) | Paul | 2007/05/13 02:40 PM |
PAE sucks (Why didn't MS take advantage of PAE?) | Peter Arremann | 2007/05/13 04:38 PM |
PAE sucks (Why didn't MS take advantage of PAE?) | Henrik S | 2007/05/14 02:31 AM |
The fragility of your argument | slacker | 2007/05/13 02:56 PM |
The fragility of your argument | nick | 2007/05/13 04:42 PM |
The fragility of your argument | Howard Chu | 2007/05/14 01:52 AM |
The fragility of your argument | Dean Kent | 2007/05/14 08:19 AM |
The fragility of your argument | anon2 | 2007/05/14 07:26 AM |
The fragility of your argument | Tzvetan Mikov | 2007/05/14 08:01 AM |
The fragility of your argument | Dean Kent | 2007/05/14 08:16 AM |
The fragility of your argument | Linus Torvalds | 2007/05/14 10:57 AM |
The fragility of your argument | JasonB | 2007/05/14 03:48 PM |
The fragility of your argument | Dean Kent | 2007/05/14 06:36 PM |
The fragility of your argument | Ricardo B | 2007/05/16 01:40 AM |
The fragility of your argument | Dean Kent | 2007/05/16 02:32 AM |
The fragility of your argument | Ricardo B | 2007/05/16 05:41 AM |
PS | Ricardo B | 2007/05/16 05:50 AM |
The fragility of your argument | Dean Kent | 2007/05/16 08:07 AM |
Modern web browsing | S. Rao | 2007/05/16 08:16 AM |
Aha! | Dean Kent | 2007/05/16 08:27 AM |
Aha! | Dean Kent | 2007/05/16 08:32 AM |
Aha! | S. Rao | 2007/05/16 09:34 AM |
The fragility of your argument | Ricardo B | 2007/05/16 09:00 AM |
The fragility of your argument | Vincent Diepeveen | 2007/05/16 09:10 AM |
The fragility of your argument | Paul | 2007/05/16 02:01 PM |
The fragility of your argument | Vincent Diepeveen | 2007/05/17 02:05 AM |
The fragility of your argument | anon2 | 2007/05/15 12:35 AM |
Splits vs page allocations? | Matt Sayler | 2007/05/15 06:33 AM |
What's your point? | Michael S | 2007/05/13 07:55 AM |
What's your point? | anonymous | 2007/05/13 10:08 AM |
What's your point? | Michael S | 2007/05/13 10:31 AM |
let's stay with x86-64 for now, please | JasonB | 2007/05/13 06:16 AM |
x864 =) (NT) | some1 | 2007/05/15 02:03 AM |
Rock/Tukwila rumors | IntelUser2000 | 2007/05/06 01:27 PM |
Rock/Tukwila rumors | m | 2007/05/13 07:05 AM |
Rock/Tukwila rumors | mas | 2007/05/15 08:40 AM |