Brendan (btrotter.delete@this.gmail.com) on January 26, 2020 10:17 pm wrote:
> Jukka Larja (roskakori2006.delete@this.gmail.com) on January 25, 2020 7:29 am wrote:
> > Brendan (btrotter.delete@this.gmail.com) on January 25, 2020 1:46 am wrote:
> > > Jukka Larja (roskakori2006.delete@this.gmail.com) on January 24, 2020 9:42 pm wrote:
> > > >
> > > > I'm not sure I understand, but if I do, I should not be able to run a program that does "malloc(15GB)"
> > > > on my system with 5 GB page file (that's not the case in case you're wondering).
> > >
> > > You shouldn't be able to "malloc(15GB)" on a system that can't increase the size of the
> > > page file up to 15+ GiB later (if/when necessary). Of course if you don't use (modify)
> > > all of the 15 GB you allocated it might never increase the size of the page file.
> >
> > Just to be extra sure (though there really wasn't any doubt really), before I wrote
> > my last comment I wrote a program that allocated 15 GBs, wrote it full of stuff and
> > exited. Commit went up by 15 GBs (at the time of allocation, it wasn't increasing gradually)
> > just as expected and page file size didn't increase, just as expected.
> >
> > If you actually use Windows with normal out-of-the box settings,
> > it is trivial to note that it doesn't work like you decribe.
>
> For default settings; you'd need to fill up your file system with normal files
> to determine how much "max. commit size" depends on "how much Windows could
> increase the page file size later if necessary". That is not as trivial.

No you don't. Just take a look at working sets of running programs and you see that they wouldn't fit in normal pagefile. It doesn't matter anyway, since I already tested it and reported the results.

> > > I'm more used to the old rule of thumb, which is "twice as much swap as you have RAM"
> > > (e.g. 64 GiB of swap on a system with 32 GiB of RAM), and your idea of "relatively large"
> > > (5 GiB vs. 64 GiB) sounds more like my idea of tiny (less than 10% of 64 GiB).
> >
> > I believe that rule of thumb hasn't been relevant for about 20 years, but I'm not
> > 100 % sure. Could have been changed as late as some Windows XP service pack.
>
> I doubt the rule of thumb was ever considered accurate for any specific case. If you assume that
> the user expects the system to cope with "RAM full of working set" (which is reasonable - more
> RAM is a waste of $$ and less RAM is a performance problem), then ideal swap size depends on the
> ratio of "working set size to memory OS committed to" which varies for different software.

I think Windows 3.0 worked by that rule and the rule then stayed in effect for far too long.

I don't think "RAM full of working set" is reasonable expectation. 90 % of time "RAM half empty" is much more reasonable, so that when there's something slightly unusual going on 10 % of the time, the computer doesn't start to lag like crazy.

> > For the usual usage patterns on laptops and desktops, large page file doesn't make sense. I'm
> > a very unusual case with hundreds of tabs open in Firefox and Chrome, couple of instant messengers
> > (each hundreds of megabytes, because of course you need a browser to make a desktop app these
> > days), Thunderbird (working set 600 MBs), LibreOffice Calc and some smaller things. All that
> > nets commit of about 16 GBs. With common usage patterns (mostly running one or two apps at a
> > time, closing the browser and all tabs after each session) you're at fraction of that.
> >
> > Then consider how small the SSDs are and it makes even less sense. For common usage patterns
> > more memory means less swap space. If user does something special, then page file will grow,
> > but it makes no sense to steal even a single gigabyte out of some 120-250 GB SSD.
>
> "Usual usage patterns" are a myth. E.g. if you ask an accountant, an aerospace engineer, an
> architect, and artist what software they use you'll get 4 very different answers. I also find
> it odd that you (a game developer) didn't include games as part of your "common" usage.

Games aren't all that memory intensive (typically. There are exceptions, just like me browser use is exceptional). Assets are mostly held in GPU memory (and if GPU doesn't have (enough) memory, then it typically doesn't have enough power to use full resolution textures anyway). Very few games require more than 8 GBs or recommend more than 16. Those numbers typically have quite a bit of air, as there's no way to know what people will have running in the background. There are actually quite a large number of people who run games on systems below minimum requirements.

According to Steam hardware survey, most common GPU memory amount is 6 GBs (20 %), more than that is less than 20 % (8 GB is about 16 %). On PS4 and Xbox One, you can use about 6 GBs, splitted in whichever way you want. If you are one of those with 500+€ GPU, you'll probably buy more RAM too. Hardware survey current has 16 GBs as most common (46 %), but just month ago it was about the same as 8 GB (now 31 %). More than 16 is about 5 %, which is about the same number (or a bit more) as GPUs with more than 8 GBs of memory.

As for "Usual usage patterns", maybe I should have emphasized more that I meant anything below certain requirements. My usage patterns are crazy compared to most people, yet I could still manage with 16 GB RAM and couple of GB page file.

> For SSD sizes, a GiB is worth about 12 cents now, so 64 GiB of "SSD space" is worth less than a single
> meal at a decent restaurant.

Sadly, people don't by SSDs by GB. I have 66 GBs free and if I want to have more, I need to buy 1 TB model and go through considerable trouble to transfer the Windows installation from old disk to new one (I can do that. Most people don't know how). When I was buying this laptop, Dell Precision somethingsomething, model with 1 TB disk wasn't available from the only consumer facing retailer that was selling these Precisions.

Go take a look at the lists of top selling laptops people are buying. You'll notice that 64, or even just 16 GB is quite a chunk of typical SSD.

> However; I still think that "tiered swap" could be significantly better economics
> (e.g. $1 of SSD space and $2 of "rotating disk" space;

No way to install rotating disk or even a second disk to most laptops, and few desktops come preconfigured in that way, so not really relevant for most of the discussion.

> > It's also pretty uncommon to need to increase page file size.
>
> I was mostly thinking about Windows' "auto-manage page file size" where it frequently increases
> page file size. Also note that (in Windows 8) I could only find "initial size + max. size"
> settings, where Windows would still increase the page file size from the initial size up to
> the max. size if you don't set the initial size equal to the max. size to prevent it.

Why would it need to frequently increase page file size? Are you still presuming all allocations must be backed by page file?

> You can resize partitions now (if you want to avoid the "backup, re-partition, reinstall/restore" option).

I'm sure you can, but it's not an operation I would do lightly.

> There's multiple reasons for multiple partitions - to enforce quotas without actually having quotas (e.g.
> to make sure "/home" can't gobble all disk space and prevent updates), for performance reasons, etc.

Yes, but that wastes space. Every single one of my nicely partitioned Linux installs ended up with some partition way too big and some other too small (typically /home being oversized and some other partitions too small after couple of upgrade cycles and accumulation of cruft). Maybe someone with more experience could have anticipated the problems. I just went with what the installers suggested.

> > If you want to backup Windows, or any OS for that matter, I'm sure there's more than one file to exclude.
> > Why would restoring old page file be a problem anyway (unless you are talking about restoring over running
> > system, which doesn't really make sense in this context, if you have any idea about how Windows works)?
>
> For restoring in abnormal situations where the OS isn't running, restoring the page file
> is a waste of time. In normal situations (whether it's "system restore" or something else)
> restoring the obsolete page file is a disaster (in addition to being a waste of time).

While system is running, you can't restore any Windows system files or registry and will have trouble with Program Files and some of the configuration files too. So complaining about pagefile, while there are probably over 100000 other files that need special handling sounds pretty useless.

If your backup is just a disk image of running system, restoring that (with some boot disk) will have Windows behave as if it has unexpectedly lost power. Pagefile doesn't cause any special trouble.

Depends on situation what sort of backup makes sense.

> > > Then there's things like RAID and whole disk encryption (do you really want "RAID-5" and encryption
> > > for your page file, just because you did want those things for normal files on "C:/"?).
> >
> > No idea what Windows does with RAID 5 page file. No idea if your C: can be RAID
> > 5 in Windows either. As for encryption, you'll probably want (all) your page
> > file(s) encrypted, if you decide system drive encryption is your thing.
>
> Windows does support RAID 5 (but not RAID 6?); and I'd expect that (as RAID
> is at a lower level than file system) your page file would also using RAID.

Does it support software RAID 5 on system disk? I couldn't install even to RAID 0 about a year ago, but it could have just been me not finding proper search terms. Hardware RAID of course is possible with appropriate driver.

> For encryption; you might want swap encrypted but not file system, or files encrypted but not swap, or you
> might want different types of encryption for different cases. My main point here is that "artificial conflation
> of file system and swap" prevents flexibility (for redundancy/RAID and encryption, and anything else).

Nothing prevents you from making a separate partition and filling it with swap file. You'll probably want to do something to Windows' "disk nearly full" spam though. Some people used to do this so they could use FAT for "swap partition" for better performance. Never saw anyone quoting any benchmarks or even saying that the system feeled snappier.

My point is that for most people, who have never even heard about a page file, separate partition is less flexible and harder to manage on the fly. When those same people decide to encrypt their system disk, it's probably a very good default to encrypt the page file too (though I think Windows encrypts page file content anyway).

-JLarja