Hi,

Jukka Larja (roskakori2006.delete@this.gmail.com) on January 25, 2020 7:29 am wrote:
> Brendan (btrotter.delete@this.gmail.com) on January 25, 2020 1:46 am wrote:
> > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 24, 2020 9:42 pm wrote:
> > > Brendan (btrotter.delete@this.gmail.com) on January 24, 2020 6:27 pm wrote:
> > > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 23, 2020 6:36 am wrote:
> > > > > Brendan (btrotter.delete@this.gmail.com) on January 22, 2020 3:28 pm wrote:
> > > > > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 22, 2020 6:12 am wrote:
> > > > > > > Brendan (btrotter.delete@this.gmail.com) on January 22, 2020 3:32 am wrote:
> > > > > > >
> > > > > > > > The right way to think about the way memory management works in Windows (unless swap is
> > > > > > > > disabled by an idiot) is to think of physical memory as nothing more than a cache of the
> > > > > > > > page file; where "max. virtual memory size = page file size". Physical memory size and
> > > > > > > > physical memory availability are irrelevant - they have no effect on OOM whatsoever.
> > > > > > >
> > > > > > > No, that's not right (you may be right for some older Windows version, but I'm pretty sure even
> > > > > > > 2000 didn't work like that). By default, Windows allocates very small page file, especially on a
> > > > > > > system with large RAM. Like I wrote in the other post, on 64 GB system, page file was 9 GB[1].
> > > > > >
> > > > > > OK; my description was a little incomplete, and could've been more like "think of physical memory
> > > > > > as nothing more than a cache of the page file; where "max. virtual memory size = page file size",
> > > > > > and where "page file size" may not be a fixed size and may grow as needed (if possible)."
> > > > > >
> > > > > > I probably should've also said "swap size" instead of "page file size" to cover non-default configurations.
> > > > >
> > > > > Still wrong. Windows is quite happy to run with page file significantly smaller than physical memory
> > > > > (running completely without page file may be stupid for unrelated reasons, of which I'm not that
> > > > > familiar with). That's probably the normal mode of operation with current memory amounts.
> > > >
> > > > Why wrong?
> > > >
> > > > I'd expect that with page file significantly smaller than physical memory Windows
> > > > would only commit to "max. swap"; and then the rest of the physical memory (that can't
> > > > be used for "cache of swap") would be used for "cache of file data" (and anything
> > > > else Windows might cache that is not part of "memory processes committed to").
> > >
> > > I'm not sure I understand, but if I do, I should not be able to run a program that does "malloc(15GB)"
> > > on my system with 5 GB page file (that's not the case in case you're wondering).
> >
> > You shouldn't be able to "malloc(15GB)" on a system that can't increase the size of the
> > page file up to 15+ GiB later (if/when necessary). Of course if you don't use (modify)
> > all of the 15 GB you allocated it might never increase the size of the page file.
>
> Just to be extra sure (though there really wasn't any doubt really), before I wrote
> my last comment I wrote a program that allocated 15 GBs, wrote it full of stuff and
> exited. Commit went up by 15 GBs (at the time of allocation, it wasn't increasing gradually)
> just as expected and page file size didn't increase, just as expected.
>
> If you actually use Windows with normal out-of-the box settings,
> it is trivial to note that it doesn't work like you decribe.

For default settings; you'd need to fill up your file system with normal files to determine how much "max. commit size" depends on "how much Windows could increase the page file size later if necessary". That is not as trivial.

> > > > Note: All files (and parts of files) mapped as "read-only" wouldn't be included in
> > > > "how much OS has committed to" (those area can be backed by file system not swap).
> > >
> > > Yes, I mentioned this earlier (our game Trine 4 maps about 14 GBs of
> > > asset files on start and that doesn't add up against commit limits).
> > >
> > > > Oh my - I did some checking and it seems that Windows doesn't even support swap partitions
> > > > (it only supports "page file on normal file system"); so it always has to worry about
> > > > normal files and swap competing for the same space, can't avoid "file system meta-data"
> > > > overhead, and has to worry about file fragmentation for the page file.
> > >
> > > Windows doesn't start with zero size page file. It starts with some relatively large one (like five GBs
> > > on my 32 GB laptop, nine on that 64 GB system I mentioned earlier) and generally expects not to need
> > > to increase the size. Page file persists over boots, so fragmentation is only an issue if one fragments
> > > the disk before page file is create (or page file size is increased a lot on a fragmented disk).
> >
> > I'm more used to the old rule of thumb, which is "twice as much swap as you have RAM"
> > (e.g. 64 GiB of swap on a system with 32 GiB of RAM), and your idea of "relatively large"
> > (5 GiB vs. 64 GiB) sounds more like my idea of tiny (less than 10% of 64 GiB).
>
> I believe that rule of thumb hasn't been relevant for about 20 years, but I'm not
> 100 % sure. Could have been changed as late as some Windows XP service pack.

I doubt the rule of thumb was ever considered accurate for any specific case. If you assume that the user expects the system to cope with "RAM full of working set" (which is reasonable - more RAM is a waste of $$ and less RAM is a performance problem), then ideal swap size depends on the ratio of "working set size to memory OS committed to" which varies for different software.

> For the usual usage patterns on laptops and desktops, large page file doesn't make sense. I'm
> a very unusual case with hundreds of tabs open in Firefox and Chrome, couple of instant messengers
> (each hundreds of megabytes, because of course you need a browser to make a desktop app these
> days), Thunderbird (working set 600 MBs), LibreOffice Calc and some smaller things. All that
> nets commit of about 16 GBs. With common usage patterns (mostly running one or two apps at a
> time, closing the browser and all tabs after each session) you're at fraction of that.
>
> Then consider how small the SSDs are and it makes even less sense. For common usage patterns
> more memory means less swap space. If user does something special, then page file will grow,
> but it makes no sense to steal even a single gigabyte out of some 120-250 GB SSD.

"Usual usage patterns" are a myth. E.g. if you ask an accountant, an aerospace engineer, an architect, and artist what software they use you'll get 4 very different answers. I also find it odd that you (a game developer) didn't include games as part of your "common" usage.

For SSD sizes, a GiB is worth about 12 cents now, so 64 GiB of "SSD space" is worth less than a single meal at a decent restaurant. However; I still think that "tiered swap" could be significantly better economics (e.g. $1 of SSD space and $2 of "rotating disk" space; especially when you're expecting that most of the swap will only be for "committed to but not modified" and therefore expecting the performance will be irrelevant); except that I'm still skeptical about "SSD longevity" claims (especially for consumer grade hardware where I think it's all based on "disk not full" statistics) and would want more "unlikely to be used" SSD space (in the form of swap space that's not expected to actually be used) to give wear leveling more freedom to level wear (and improve longevity, and reduce replacement costs).

> > Yeah, I get the impression that the "auto-increase page file size' in Windows isn't great (I'd prefer
> > swap partitions; and can't see why it doesn't make processes wait while it increases page file size
> > and can't be better at leaving more of page file's disk space allocated across reboots).
>
> I think the algorithm tries to increase page file size beforehand, but when something
> like 32^32 compile processes get launched during a three second window or some shader
> compiler just decides it needs 28 GBs for something, it's hard to keep up.
>
> As long as I've used SSDs, I haven't actually seen the Windows warning popup on my own systems, even
> when page file size has been increased. I've only seen it on our build servers, which do experience
> those extreme cases of memory consumption increases. So maybe the page file increase algorithm is
> actually good enough for most users and MS just expects server people to set the size by hand.
>
> That said, my experiment with 15 GB memory allocation pushed the total commit to bit over 34 GBs (was running
> Visual Studio at the time too), while the limit (currently, with that 5 GB page file still in place) is 38.
>
> Just for science I tried allocating 25 GBs. The allocation took a long time, but there was no warning popup from
> Windows. The program then run quickly (it just touched every 2048th byte in allocation) and quit. To my surprise,
> the page file was now 8 GBs. I run the program again and monitored page file more closely. It was actually 16
> GBs during program run but was immediately (within a second or two) reduced to 8 GBs after the run.
>
> So it seems Windows is actually pretty good in this runtime page file resizing. Not sure
> whether that's because the SSD in this laptop is quite a bit faster than anything in our
> servers or if there's been some recent change in how the size increases are handled.
>
> Out of interest, I also tried larger allocations. 75 GB one failed, even though there
> would have been about 13 GBs free space on disk after appropriate page file size increase.
> 70 GB one worked, and system commit went to about 90 GBs as expected.
>
> Since that was so fun, I decided to try allocating that 70 GBs 2 kBs at a time. It went fine close to the
> end. I was monitoring the page file size with dir /Ahs and. It was gradually being increased (in rather
> small increments) until I got: "Ei riittävästi muistiresursseja komennon käsittelyä varten."
>
> (That's "not enough memory resources to handle the command") and after another try simply "Out
> of memory." (no more localization, eh?). At that point, screen went black (not sure what crashed),
> Visual Studio crashed, Firefox crashed, Thunderbird crashed, Process Explorer crashed and something
> called "Dell Client Management Service" crashed too (seen in Event Viewer. I don't even know what
> that is). And I still didn't get that popup from Windows, though according to Event Viewer, there
> was an Application Popup with the familiar text in it. Maybe it crashed too.
>
> Nice detail about Firefox: I didn't lose any of the text I had written here so far.
>
> > > As for "avoiding file system meta-data", why wouldn't Windows be able to do that?
> > > It's not like NTFS needs to be aware of what goes on inside a page file.
> >
> > When the OS wants to access data at "offset in page file" it'd have to convert that into "offset
> > in partition" (while taking into account page file fragmentation, etc) instead of just doing
> > a fast/simple addition ("offset in partition = start of partition + offset in swap").
>
> Each fragment could be treated as separate page file.

With separate file system meta-data (like "C://pagefile000.sys", "C://pagefile001.sys", ...) to keep track of each individual "unfragmented extent"?

> And why use offset in partition
> instead of offset on disk (if such optimization really makes any difference).
>
> > Also; increasing the page file size would involve finding free blocks of disk space and marking
> > them as "not free" in whatever NTFS uses to keep track, then adding them to whatever NTFS uses
> > to keep track of which parts of a disk are used by a file, plus updating the directory entry
> > (e.g. file size field) and any other accountancy/statistics (e.g. free space stats).
>
> You talk like the paging system needs to do all that without help from NTFS. It's not like MS needs
> to even support a page file on every file system, for example, on ReFS it's not supported.
>
> It's also pretty uncommon to need to increase page file size.

I was mostly thinking about Windows' "auto-manage page file size" where it frequently increases page file size. Also note that (in Windows 8) I could only find "initial size + max. size" settings, where Windows would still increase the page file size from the initial size up to the max. size if you don't set the initial size equal to the max. size to prevent it.

> > My main problem with "page file" is what happens when you reach
> > an "almost full" state - it creates a bizarre/unintuitive
> > relationship where (e.g.) allocating memory reduces the space you have left for files, and creating files
> > reduces how much memory you can allocate. Of course this problem goes away if you have a fixed size page
> > file, but then what's the point of not having a fixed size swap partition instead?
>
> With file, you can decide later. You can also change the fixed size later. From my point of view,
> the question is why have any more partitions than necessary at all? Partitions are unflexible.

You can resize partitions now (if you want to avoid the "backup, re-partition, reinstall/restore" option).

There's multiple reasons for multiple partitions - to enforce quotas without actually having quotas (e.g. to make sure "/home" can't gobble all disk space and prevent updates), for performance reasons, etc.

I also think it's sad that there isn't any "common swap partition standard" (e.g. as part of UEFI's GPT) so that when you have multiple operating systems installed (e.g. dual boot between Windows and Linux) they can't both use the same swap partition/s.

> I have a Linux box with 10 GB swap partition and 32 GBs of memory, because that computer started
> with just 8 and resizing partition is too much work for me. It's likely to be 10 + 64 or 10 +
> 128 in the future, when I update the hardware and just clone the disk or use it as is.
>
> > There'd also be work-arounds in various utilities (doing a back-up of the file system?
> > Better add a special case to avoid backing up the page file. Trying to restore from a
> > checkpoint? Better not try to restore the old page file. Want to search for all files
> > modified in the last 20 minutes? Better hide the page file in the search results.)
>
> If you want to backup Windows, or any OS for that matter, I'm sure there's more than one file to exclude.
> Why would restoring old page file be a problem anyway (unless you are talking about restoring over running
> system, which doesn't really make sense in this context, if you have any idea about how Windows works)?

For restoring in abnormal situations where the OS isn't running, restoring the page file is a waste of time. In normal situations (whether it's "system restore" or something else) restoring the obsolete page file is a disaster (in addition to being a waste of time).

> > Then there's things like RAID and whole disk encryption (do you really want "RAID-5" and encryption
> > for your page file, just because you did want those things for normal files on "C:/"?).
>
> No idea what Windows does with RAID 5 page file. No idea if your C: can be RAID
> 5 in Windows either. As for encryption, you'll probably want (all) your page
> file(s) encrypted, if you decide system drive encryption is your thing.

Windows does support RAID 5 (but not RAID 6?); and I'd expect that (as RAID is at a lower level than file system) your page file would also using RAID.

For encryption; you might want swap encrypted but not file system, or files encrypted but not swap, or you might want different types of encryption for different cases. My main point here is that "artificial conflation of file system and swap" prevents flexibility (for redundancy/RAID and encryption, and anything else).

> > I think ReadyBoost was to improve latency for normal files (especially
> > during boot where OS is trying to fetch all drivers, GUI, etc).
>
> I thought it worked for page file too, but I could be wrong. It hasn't been relevant since 2011 for me.

Maybe; but that (using SSD to cache swap on slow drive instead of just having 2 swap providers) would seem silly to me.

- Brendan