Hi,

nick (anon@anon.com) on 5/15/06 wrote:
>Brendan (btrotter@gmail.com) on 5/14/06 wrote:
>>nick (anon@anon.com) on 5/14/06 wrote:
>
>>>Doesn't matter, if they have to both take the lock ->
>>>cacheline bouncing; if one CPU writes a messages which is
>>>read by another -> cacheline bouncing.
>>
>>That would involve a cache line fill. It's not even close to the performance effects
>>of lock contention (which either involves repeated cache
>
>No. I told you, you're 10 years out of date. Lock
>contention is largely not an issue any more, it is cache
>misses "cache line fill". Read some of literature for
>reasons why there is increasing focus on lock-free
>algorithms and data structures.

So you tell me what's worse, cacheline bouncing caused by a message queue lock and nothing else, or cacheline bouncing caused by a device driver's lock in addition to more cacheline bouncing caused by modifying the device driver's internal data?

Of course theoretically the device driver's internal data might be lock-free or wait-free, but theoretically so could the message queue (and it'd probably be easier to develop one lock-free or wait-free algorithm for IPC than it would be to develop one or more for every device driver).

>You're first assuming that there is a large amount of
>lock contention in scalable OSes then concluding
>that, therefore, the relatively minor cost of cacheline
>contention is unimportant.

I assume that lock contention should be avoided in scalable OSs, and that cacheline contention is unimportant until lock contention is avoided. If minimizing lock contention wasn't important then we wouldn't be discussing the problems of a "big kernel lock".

>>Still, I'm getting "side-tracked". My original comment was about the time taken
>>by the longest kernel function. Consider a system where a CPU spends 50% of it's
>>time in the kernel on average, compared to a system where a CPU spends 1% of it's
>>time in the kernel on average. For the first system "big kernel lock contention"
>>is extremely likely (even for a 2-way system), while for the second system "big
>>kernel lock contention" is unlikely (even for 8-way SMP).
>
>I'm not talking about lock contention, I'm talking about
>cacheline contention. But if all else was equal, of course
>you're right.
>
>However, your 1% system may need to enter the kernel much
>more *frequently*, in which case the cacheline contention
>might be as bad or worse.

I'd suggest that it doesn't matter if that 1% is caused by entering the kernel 10 times (at 0.1% per time) or if it's caused by entering the kernel twice (at .5% per time) - it's still 1% regardless of how frequently you enter the kernel.

What you might be trying to say is that the CPU might actually spend 2% of it's time in kernel space because the kernel is entered more often, in which case you'd be right (but I don't see how this makes a fundamental difference). In general, if the time spent inside a micro-kernel is equal to the amount of time spent inside a monolithic kernel, then either the micro-kernel designer needs to be shot (or the monolithic kernel designer deserves several awards for outstanding acheivements).

>>For monolithic systems the time spent in the kernel includes the time spent within
>>all device drivers, so the time spent inside the kernel is going to be greater than
>>an equivelent micro-kernel (where time spent in the kernel doesn't include the time
>>spent within device drivers). Therefore using a big kernel lock is going to effect
>>performance/scalability in a monolithic system much more than it would effect performance/scalability
>>in an equivelent micro-kernel.
>
>Yes. But nobody would call a monolithic big lock kernel
>scalable, so I don't think you can transitively conclude
>anything about the microkernel.
>
>>>In Linux, the control data does not need to go across
>>>node because it is basically kept in the stack of the
>>>running process. Instruction text is a different matter,
>>>but AFAIK, that can be replicated on the big NUMA machines
>>>in Linux.
>>
>>My last prototype used multiple copies of the kernel (one for each NUMA domain)
>>and my current prototype uses multiple copies of the kernel and multiple copies
>>of "static kernel data" (anything that doesn't change after boot). My kernels are
>>normally less than 64 KB though so the memory costs aren't too bad and it's still
>>sensible when the "across node penalty" isn't too high (e.g. 2 or 4 way Opterons).
>>Linux is typically around 3 MB so you might get better results using this memory for disk cache instead. :-)
>
>Well are you replecating the text of your servers?

No - they are independant processes that use CPU affinity to ensure that they are always run on the same NUMA domain...

>Considering that nobody in Linux even cares that much
>about it except the guys with 1024 CPU systems, I'm
>guessing it is completely unmeasurable on your kernel
>(outside microbenchmarks, maybe). :-)

Given that both AMD and Intel are increasing the number of cores rather than increasing core frequency (and that I predict Intel will be shifting to something like hypertransport/NUMA in the near future), the number of people who care about it is probably going to increase a lot by the time it matters to me.

My work consists of a series of prototypes, where each prototype builds on the last. The newest prototype uses a "modular micro-kernel", is 32 bit and 64 bit and is designed to scale to large NUMA systems. I've basically reached the end of the series of prototypes (there's nothing left to add and the worst of the bottlenecks are gone). With some luck, my current prototype will become the basis for an OS. I'm expecting it to take another 3 years before I've got a bare working system running on legacy hardware, but it's too different to port applications (or drivers) to it and it'll probably take 10 years or more before it's actually usable. I knew this before I started, which is why I've spent so much time making sure the kernel design is "right".

Anyway, real world benchmarks (like comparing web server and database performance) is a long way off...

>I'd wager that passing messages across interconnect would
>be more interesting...

Passing messages between NUMA domains will be slower, but it's designed for passing messages across a LAN so I doubt NUMA domain boundaries are going to matter much in comparison.


Cheers,

Brendan