I_vs (ivs.delete@this.anon.com) on November 24, 2020 11:12 pm wrote:
> Maynard Handley (name99.delete@this.name99.org) on November 24, 2020 2:06 pm wrote:
> > I_vs (ivs.delete@this.anon.com) on November 24, 2020 11:40 am wrote:
> > > Gabriele Svelto (gabriele.svelto.delete@this.gmail.com) on November 23, 2020 12:56 pm wrote:
> > > > If you want microbenchmarks they aren't hard to find either:
> > > >
> > > > https://www.bitsnbites.eu/benchmarking-os-primitives/
> > >
> > > This is an interesting one! My M1 MacBook Pro has just arrived so I have just run this
> > > suite on it. Here are the results (only one run, just to get a general idea):
> > >
> > > create_threads 6.08 us /thread
> > > create_processes 83.18 us /processs
> > > launch_programs 218.5 us /program
> > > create_files 101 us/file
> > > mem_alloc 82.139015 ns / alloc
> > >
> > > It seems that M1 brings significant improvements across the board. My 16" Intel
> > > MacBook Pro was basically as slow as the Mac Mini results in that blog post.
> > >
> > > I am particularly curious how Linux manages to be this fast on file creation.
> > > It seems like it never hits the drive, just some sort of in-memory cache?
> >
> > Most of the difference boils down to
> > - different functionality
> > - different ideas of what is acceptable safety
> >
> > Functionality includes, in particular, fsevents and the associated machinery tracking changes,
> > and thus powering both Spotlight (system search) and TimeMachine (file backups). This overhead
> > varies depending on exactly how you've configured macOS and what file system you're using.
> >
> > Acceptable safety boils down to different opinions about just what is promised (how much persistence,
> > with what ordering guarantees) by various explicit calls and implicit behavior.
> > Apple's take on this has long been that you cannot JUST persist to the extent that is theoretically
> > legal as promised you vendors (in particular drive vendors) because THEY lie about when stuff has really
> > been persisted (so that they can look good in benchmarks), you have to go beyond that. Presumably this
> > fairly draconian stance (which, yes, has serious performance implications) can be avoided once Apple
> > controls the entire drive (ie for flash being controlled by an Apple SSD controller).
> > The Linux stance on this seems to vary, depending on to whom you talk, from "of course you can
> > trust the HW guys, their firmware is the greatest" to "OK, yes, admittedly things CAN go wrong
> > in theory, but it's just not worth the performance hit to worry about something so unlikely".
> >
> > The last time I cared enough to look at this in detail,
> > the state of the art was essentially as described here:
> > https://bugs.mysql.com/bug.php?id=56550
> > (scroll down to 9 Nov 2010)
> >
> > Now has Apple changed since then? My guess is probably on pure Apple SSDs and nowhere else.
> > Has Linux changed since then? Not a clue.
> > Has drive firmware changed since then? Well, benchmarks are still a thing,
> > so my guess is the incentives remain what they were in 2010...
> >
> >
>
> Just a quick comment on this: I run the create_files microbenchmark again and noticed that there was a big spike
> in activity from fseventsd and mds. If macOS automatically indexes metadata and persistently tracks events of
> each file on creation, then yes, it would make perfect sense that file creation is significantly slower.
>
> When I use ionotifywait to log changes to the test on Linux, file creation times go from
> 9 us/file to 70 us/file (using a ramdisk). Linux is still faster of course, but I suppose
> that always-on event logging accounts for much of the observed difference.

Good result, that's a bit closer to what I expect, although I would guess Linux may be closer to 2x faster if you ran on the same CPU (a single threaded non-FP irregular code test will be hitting max turbo frequency or very close to it on both CPUs, which would put your Linux result at 50us with a 3.7/5.0 ratio. So that was actually a good suggestion from Maynard. Although I still disagree with him in that I don't believe OSX is getting any special advantage from its unsafe data integrity defaults here like it does in some other tests involving fsync().

Of course we still don't exactly know if we are doing an Apples to apples (/ducks) comparison here because we don't really know the extent of these notification features or other possible things going on. It would almost be better if you could turn off those fancy things on OSX and compare again, but maybe that's not possible.

I suspect other things like "process creation" may be also doing some fundamentally different work between the systems, possibly some security check or call to an AV process or logging/auditing or who knows what may also be slowing down results somewhat. Certainly it's not unusual to hear on Windows if AV or some particular service or weird driver is turned off or tweaked then it can turn from a slug to usable. I'm sure Windows is not 2 orders of magnitude slower on fork+exec when things are running properly and comparably.

I certainly don't think any kind of microkernel no matter how bad should be responsible for orders of magnitude slow down here - a synchronous single-threaded test like this is about the least problematic case for a microkernel (even if you had a pure microkernel that required a few full context switches into different kernel services, those should be on the order of a few microseconds even if you have to switch address spaces so that doesn't fundamentally explain +50 microseconds) -- it's when you get into scaling and pipelining and concurrency that you can see bigger breakdowns (and which is why they've shoehorned most of the IO subsystem into a BSD kernel on the side I would say). But I digress.

A big part of the problem for end users even somewhat technical people are these defaults which don't suit what they want to do, and there tends to be no obvious indication of the reason for the slow performance or how to fix it. Although some might counter stack exchange is fine for such FAQ these days.