Brett (ggtgp.delete@this.yahoo.com) on November 26, 2020 11:29 am wrote:
>
> The beauty of this approach is that all this database needs is to import/export git, and
> have a filesystem driver, and you don’t care what the rest of the world is doing.
>
> You don’t even need git, git can write to the filesystem, there are just some opportunities
> to optimize git while you are at it. You can inline compression if it is free, etc.
>
> Everything just works, and you can now optimize the speed of builds and git by tuning the database
> layout behavior. A folder tree with hundreds of files spread all over the disk is now a small
> set of large linear packed files that stream quickly and use less disk and less ram.

You're forgetting about fragmentation, which will inevitably happen as you modify the sources. And if we're talking about SSDs then fragmentation is not an issue in either case.

> This is a custom filesystem just for compiling, forcing bulk packed loads for speed. As a single purpose filesystem
> it can track usage and reconfigure to optimize usage. Something ordinary filesystems cannot do.

I don't see why. In fact, I don't see why that should be a filesystem's business in the first place. Rather, it should be done by IO scheduler regardless of the underlying filesystem. And, if I'm not mistaken, BFQ is doing some optimizations along these lines, though not specifically for compiling.

And my main point stands - what you're saying sounds pretty much like yet another filesystem, possibly with some IO scheduler tuning, which is nothing revolutionary. At this point I see no evidence that it will significantly outperform existing filesystems, unless they are inherently inefficient for this kind of workload on some particular hardware. Designing a filesystem is no small feat, and at this point it is difficult to significantly improve upon the existing ones. Case in point BTRFS.

> Important people read this site, someone may implement this idea, and as
> a software developer this will make me and you happy with faster builds.
>
> Because Microsoft and Apple are not going to optimize their OS’s for mmaping thousands of files like Linus
> does. In the end the Linus way will lose the performance crown. Everyone else has abandoned filesystems for
> databases, it is about time programmers did. Insert quote about the cobbler’s children do not wear shoes.

Databases, especially relational ones, do not map well onto filesystems. Primarily, because (existing) filesystems are mostly aimed at storing data, not necessarily searching it. Various indexing services exist precisely to mitigate this.

You don't need searching for compiling sources. In fact, you only need efficient open/read/write/close operations, which is perfectly in line with what current filesystems do.