Brendan (btrotter.delete@this.gmail.com) on January 25, 2020 1:46 am wrote:
> Hi,
>
> Jukka Larja (roskakori2006.delete@this.gmail.com) on January 24, 2020 9:42 pm wrote:
> > Brendan (btrotter.delete@this.gmail.com) on January 24, 2020 6:27 pm wrote:
> > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 23, 2020 6:36 am wrote:
> > > > Brendan (btrotter.delete@this.gmail.com) on January 22, 2020 3:28 pm wrote:
> > > > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 22, 2020 6:12 am wrote:
> > > > > > Brendan (btrotter.delete@this.gmail.com) on January 22, 2020 3:32 am wrote:
> > > > > >
> > > > > > > The right way to think about the way memory management works in Windows (unless swap is
> > > > > > > disabled by an idiot) is to think of physical memory as nothing more than a cache of the
> > > > > > > page file; where "max. virtual memory size = page file size". Physical memory size and
> > > > > > > physical memory availability are irrelevant - they have no effect on OOM whatsoever.
> > > > > >
> > > > > > No, that's not right (you may be right for some older Windows version, but I'm pretty sure even
> > > > > > 2000 didn't work like that). By default, Windows allocates very small page file, especially on a
> > > > > > system with large RAM. Like I wrote in the other post, on 64 GB system, page file was 9 GB[1].
> > > > >
> > > > > OK; my description was a little incomplete, and could've been more like "think of physical memory
> > > > > as nothing more than a cache of the page file; where "max. virtual memory size = page file size",
> > > > > and where "page file size" may not be a fixed size and may grow as needed (if possible)."
> > > > >
> > > > > I probably should've also said "swap size" instead of "page file size" to cover non-default configurations.
> > > >
> > > > Still wrong. Windows is quite happy to run with page file significantly smaller than physical memory
> > > > (running completely without page file may be stupid for unrelated reasons, of which I'm not that
> > > > familiar with). That's probably the normal mode of operation with current memory amounts.
> > >
> > > Why wrong?
> > >
> > > I'd expect that with page file significantly smaller than physical memory Windows
> > > would only commit to "max. swap"; and then the rest of the physical memory (that can't
> > > be used for "cache of swap") would be used for "cache of file data" (and anything
> > > else Windows might cache that is not part of "memory processes committed to").
> >
> > I'm not sure I understand, but if I do, I should not be able to run a program that does "malloc(15GB)"
> > on my system with 5 GB page file (that's not the case in case you're wondering).
>
> You shouldn't be able to "malloc(15GB)" on a system that can't increase the size of the
> page file up to 15+ GiB later (if/when necessary). Of course if you don't use (modify)
> all of the 15 GB you allocated it might never increase the size of the page file.
>
> > > Note: All files (and parts of files) mapped as "read-only" wouldn't be included in
> > > "how much OS has committed to" (those area can be backed by file system not swap).
> >
> > Yes, I mentioned this earlier (our game Trine 4 maps about 14 GBs of
> > asset files on start and that doesn't add up against commit limits).
> >
> > > Oh my - I did some checking and it seems that Windows doesn't even support swap partitions
> > > (it only supports "page file on normal file system"); so it always has to worry about
> > > normal files and swap competing for the same space, can't avoid "file system meta-data"
> > > overhead, and has to worry about file fragmentation for the page file.
> >
> > Windows doesn't start with zero size page file. It starts with some relatively large one (like five GBs
> > on my 32 GB laptop, nine on that 64 GB system I mentioned earlier) and generally expects not to need
> > to increase the size. Page file persists over boots, so fragmentation is only an issue if one fragments
> > the disk before page file is create (or page file size is increased a lot on a fragmented disk).
>
> I'm more used to the old rule of thumb, which is "twice as much swap as you have RAM"
> (e.g. 64 GiB of swap on a system with 32 GiB of RAM), and your idea of "relatively large"
> (5 GiB vs. 64 GiB) sounds more like my idea of tiny (less than 10% of 64 GiB).
>
> > As I wrote earlier, when Windows needs to increase page file size, it doesn't happen immediately. Some
> > allocations will fail and Windows will even popup a dialog box explaining this. I'm not sure if page
> > file size is ever decreased on the fly, but it will be on boot. I'm not sure if the automatic algorithm
> > is smart enough to start with bigger page file, if system often runs out of commit space. On computers
> > where this matters (build servers at work), I'll just set fixed page file size that's "large enough"
> > (if it runs out, the system is either leaking or has become completely unusably laggy anyway).
>
> Yeah, I get the impression that the "auto-increase page file size' in Windows isn't great (I'd prefer
> swap partitions; and can't see why it doesn't make processes wait while it increases page file size
> and can't be better at leaving more of page file's disk space allocated across reboots).
>
> > As for "avoiding file system meta-data", why wouldn't Windows be able to do that?
> > It's not like NTFS needs to be aware of what goes on inside a page file.
>
> When the OS wants to access data at "offset in page file" it'd have to convert that into "offset
> in partition" (while taking into account page file fragmentation, etc) instead of just doing
> a fast/simple addition ("offset in partition = start of partition + offset in swap").
>
> Also; increasing the page file size would involve finding free blocks of disk space and marking
> them as "not free" in whatever NTFS uses to keep track, then adding them to whatever NTFS uses
> to keep track of which parts of a disk are used by a file, plus updating the directory entry
> (e.g. file size field) and any other accountancy/statistics (e.g. free space stats).

Not to forget that files on NTFS can have compression and encryption flag set and this applies also to pagefiles. So if the pagefile has been compressed (compresses really well) then allocation in them require decompression/compression during access. I remember having tested that on NT4 more than 20 years ago, it worked but it was not pretty performance wise.

>
> > Personally, I've always thought that swap partitions are idiotic from user perspective. OS may get
> > a tiny performance boost out of using them, but it's very inflexible system that has often caused
> > me trouble (OS installer allocated too small partition). Which is probably why it is pretty trivial
> > to add a file system based page file in most Linux distros, though not on every file system.
>
> My main problem with "page file" is what happens when you reach an "almost full" state - it creates a bizarre/unintuitive
> relationship where (e.g.) allocating memory reduces the space you have left for files, and creating files
> reduces how much memory you can allocate. Of course this problem goes away if you have a fixed size page
> file, but then what's the point of not having a fixed size swap partition instead?
>
> There'd also be work-arounds in various utilities (doing a back-up of the file system?
> Better add a special case to avoid backing up the page file. Trying to restore from a
> checkpoint? Better not try to restore the old page file. Want to search for all files
> modified in the last 20 minutes? Better hide the page file in the search results.)
>
> Then there's things like RAID and whole disk encryption (do you really want "RAID-5" and encryption
> for your page file, just because you did want those things for normal files on "C:/"?).
>
> > > For Windows, in theory it might support "tiered swap" (e.g. by benchmarking swap
> > > providers in a "zero configuration, auto-detection" way); but it's easier to assume
> > > Windows doesn't support it at all (just treats all swap providers as equal).
> >
> > Windows did have the "ReadyBoost" system back in Vista days. The idea was to use low-latency
> > (compared to an HDD) flash device (typically USB stick) to reduce page in lag. If I recall
> > correctly (and if the articles I read got it at all right), everything was always written
> > to proper page file, and ReadyBoost device was only used as a cache.
>
> I think ReadyBoost was to improve latency for normal files (especially
> during boot where OS is trying to fetch all drivers, GUI, etc).
>
> > So I wouldn't be too surprised if under the hood Windows supported tiered swap
> > and even more complex things, but Microsoft just doesn't feel like exposing them
> > (and thus requiring to actually properly test them and support them).
>
> That sounds quite plausible to me.
>
> - Brendan
>