Brendan (btrotter@gmail.com) on 5/15/06 wrote:
---------------------------

>>However, your 1% system may need to enter the kernel much
>>more *frequently*, in which case the cacheline contention
>>might be as bad or worse.
>
>I'd suggest that it doesn't matter if that 1% is caused by entering the kernel
>10 times (at 0.1% per time) or if it's caused by entering the kernel twice (at .5%
>per time) - it's still 1% regardless of how frequently you enter the kernel.

Not if each time you enter the kernel you send a message.
Most of the cache misses won't get eaten by the kernel, but
by the server which is reading the messages off the queue.

>
>What you might be trying to say is that the CPU might actually spend 2% of it's
>time in kernel space because the kernel is entered more often, in which case you'd

No.

>be right (but I don't see how this makes a fundamental difference). In general,
>if the time spent inside a micro-kernel is equal to the amount of time spent inside
>a monolithic kernel, then either the micro-kernel designer needs to be shot (or
>the monolithic kernel designer deserves several awards for outstanding acheivements).
>

Of course, but the micro-kernel is obviously directly
related to the time spent in the servers too (eg. the
number of cache misses they take); and considering ukernels
are no faster than monolithic kernels, the combination of
kernel + servers is going to be at least as much as a
monolithic kernel.

>>Well are you replecating the text of your servers?
>
>No - they are independant processes that use CPU affinity to ensure that they are always run on the same NUMA domain...
>

So you don't use shared libraries, or share program text?
OK, now you've lost the same amount of memory as Linux with
text replication.

Do you use threads of a single memory space running on
different nodes?

>>Considering that nobody in Linux even cares that much
>>about it except the guys with 1024 CPU systems, I'm
>>guessing it is completely unmeasurable on your kernel
>>(outside microbenchmarks, maybe). :-)
>
>Given that both AMD and Intel are increasing the number of cores rather than increasing
>core frequency (and that I predict Intel will be shifting to something like hypertransport/NUMA
>in the near future), the number of people who care about it is probably going to
>increase a lot by the time it matters to me.
>

Wrong. Number of cores has nothing to do with it, and
desktops/workstations/small servers will never care much
about NUMA issues because there just aren't enough sockets
to make a difference. Improvement on even an 8 socket
Opteron is probably unmeasurable on Linux, for example.

The systems I'm talking about have local/remote latency
ratios of 10:1, and going from one end of the interconnect
to the other takes ~8 router hops over probably 20 or more
meters.

*Those* guys are just starting to care about it a little
bit. And not so much because the slowdown is noticable for
the nodes taking icache faults from remote memory, but
because the combined effect of all of them saturates node0's
interconnect.

>My work consists of a series of prototypes, where each prototype builds on the
>last. The newest prototype uses a "modular micro-kernel", is 32 bit and 64 bit and
>is designed to scale to large NUMA systems. I've basically reached the end of the
>series of prototypes (there's nothing left to add and the worst of the bottlenecks
>are gone). With some luck, my current prototype will become the basis for an OS.
>I'm expecting it to take another 3 years before I've got a bare working system running
>on legacy hardware, but it's too different to port applications (or drivers) to
>it and it'll probably take 10 years or more before it's actually usable. I knew
>this before I started, which is why I've spent so much time making sure the kernel design is "right".
>
>Anyway, real world benchmarks (like comparing web server and database performance) is a long way off...

So how do you know you've done it right? Are you designing
based on assumptions, or real testing? Is this work public?

Kudos for trying, but it still doesn't sound convincing. K42
claims to be a microkernel, and occasionally they get really
excited about finding somewhere that Linux doesn't scale too
well at, and beat it. Which obviously turns out to be a
place that nobody ever cares about anyway.

>
>>I'd wager that passing messages across interconnect would
>>be more interesting...
>
>Passing messages between NUMA domains will be slower, but it's designed for passing
>messages across a LAN so I doubt NUMA domain boundaries are going to matter much in comparison.

Yeah, if it is so heavyweight that you don't notice these
cache misses, then doesn't sound like it is appropriate for
closely coupled NUMA interconnects.