Gabriele Svelto (gabriele.svelto.delete@this.gmail.com) on November 30, 2020 1:05 pm wrote:
> Louis Gerbarg (l.delete@this.g.com) on November 29, 2020 5:23 pm wrote:
> > I actually agree with that, but I don't think that changes
> > my main point. For 25+ years Linux has been focused
> > on workloads like traditional project builds where the
> > speed of spawning small processes is significant, and
> > it does it very well. The fact that people are moving to
> > modern build systems that do not lean on that strength
> > does not change the fact that pressure from those traditional workloads pushed the Linux kernel to focus on
> > making process creation cheap, just like pressure from
> > running longer lived graphical processes pushed macOS
> > and Windows to develop sophisticated dynamic linking and loading
> > technologies in order to reduce memory footprint
> > and speed up app launch times that were dominated by linking (as opposed to process creation).
>
> I haven't seen numbers to back that up but even if it was true it doesn't explain why Linux is also
> faster at a number of common OS primitives such as filesystem access, IPC, threading, pipes, sockets,
> etc... And that's without mentioning scalability. I did my comparison with macOS using a limited number
> of cores that's around Apple's sweet spot but I'm pretty sure the comparison would become increasingly
> embarassing as the number of cores involved increases. All those primitives are not only faster on
> Linux but they scale a lot better, and that's especially true for filesystem access.

I completely believe that many primitives are faster on Linux. I was only stating that macOS (and likely Windows, though I have not looked into it) load and bind system libraries faster than Linux, and that just like how fast process creation biases how people architect their systems, so do cheap dynamic libraries.

For a concrete example, I ran sccache on macOS, stopped it before it after it had "mapped" in all of the system libraries it needs. There are 144 binaries loaded (the dynamic linker, the main executable, and 142 libraries larger consisting of the implementation of Libc, pthread, etc). Despite having so many loaded images, the dynamic linker made exactly 2 calls to mprotect() and 0 calls to mmap() before it started jumping into the loaded libraries to run their static initializers. They way we load system libraries is fast because it is architected substantially differently than Linux, and it causes us to build our system with A LOT more dynamic libraries. What you probably think of as critical path primitives for dynamic loading are barely even called.

I suspect if you were to do a similar experiment on Linux you would find it makes many more calls to mprotect() and mmap(), despite probably only have a handful of dynamic libraries loaded. That means the speed of those calls is probably very important in order to quickly launch processes. I also think despite those faster primitives, if glibc was fractured into 142 dynamic libraries it would take a lot more ram and substantially slow down your process initialization times. It would work, just pretty poorly, so you probably should not do that ;-)

Is my point that macOS is somehow really magically faster for your workload and your benchmarks are wrong? Absolutely not! My point is the systems are genuinely tuned for very different workloads, and that causes the critical paths for many common taks to be quite different. It also means when you move compatible code between the systems it may behave poorly because what you think of as being critical path primitive actually isn't on some other software stack. IOW, the software design is biased toward the system it is developed on, and when you run it on a different (put compatible) system with different optimization points it may perform poorly.