Andrey (andrey.semashev.delete@this.gmail.com) on November 27, 2020 3:25 am wrote:
> Brett (ggtgp.delete@this.yahoo.com) on November 26, 2020 4:11 pm wrote:
> >
> > Git updates are largely a write only operation
>
> Even if we allow it, writes are what creates fragmentation.
>
> > editor changes only effect a handful of files and are irrelevant.
>
> No, they are not.
>
> > The database only holds code, compiler output goes to the filesystem.
>
> So now you're optimizing only source file reads? Is this really what bottlenecks the compilation?

Yes link time is a major subset of compile time, let’s solve one miracle at a time instead of demanding the moon and sun, and saying I failed because I did not deliver the moon and sun.

> > Detecting disk idle and rewriting the database for speed is a given.
>
> This has been happening with regular filesystems for decades, at least on Windows.
>
> > Detecting a compile is trivial, saving the access order so
> > as to rewrite the file order packing to match is trivial.
>
> I'm not convinced about this. Saving anything is another write. The order of reading
> can be different between runs and compilers. Think parallel compilation.

Exact order is irrelevant when you are prefetching loads ahead of time, which this database can and will do.
The files are packed and you are going to load blocks of say 64k at once.
That is about 8 files, so the first four block loads will make your 32 core chip happy, and the database will keep loading blocks ahead of use.

Compare that to mapping 4K at a time as asked for, all over a file directory on spinning rust.

> > Grep does not care about file order generally, gets a database
> > directory file list in compile order, which is near optimal.
> > Actual file order will go in and out of subdirectories almost randomly
> > but a custom grep executable can follow database file order.
>
> So now we need a custom grep as well?
>
> > The database knows which files are redundantly read from the last compile and keeps those sectors mapped.
>
> That's what filesystem block cache is for.
>
> > Now are you starting to see the benefits?
>
> No, not really.
>
> > A simple smart database turns a thousand different reads into almost one linear fire hose of packed
> > data.
>
> You still get the same thousand reads as long as you use POSIX API for them. You may optimize the underlying
> implementation of these reads, which is what those OS components I mentioned earlier are for.

This database can preload files on build detection, something a filesystem will never do.
The files are packed and you are going to load blocks of say 64k at once.

> > The Linux compile advantage of 2X over MacOS and 4X+ over Windows turns into a deficit.
> >
> > What if the database gets corrupt?
> > sudo rm -rf/*
> > git
> >
> > Yes I included a typo.
> >
> > The Linux compile advantage is a stupid pet trick with little real world
> > use outside of compiles, and will one day be rendered obsolete.
>
> Yeah, right, because compiles are not "real world" enough. The only thing that has become obsolete for
> me, as a developer, is Windows. And Mac OS has never become relevant, although for other reasons.
>