Hi,

nick (anon@anon.com) on 5/13/06 wrote:
>Brendan (btrotter@gmail.com) on 5/13/06 wrote:
>>nick (anon@anon.com) on 5/13/06 wrote:
>
>>>Great now your scalability sucks, and you can do this with
>>>a monolithic kernel anyway as Linus points out.
>>
>>The message queues are only locked when messages are being added or removed. I
>>guess it does depend on how the messaging is done - synchronous messaging would probably suck for scalability.
>
>No, it would suck. You're about 10 years behind: lock
>contention doesn't hurt scalability any more, cacheline
>contention does.

Am I?

For my last prototype, each thread has a "thread control block" containing it's message queue lock (one queue per thread). While each lock wasn't alone in it's own 128 byte area (Intel's recommendation) the data around it was deliberately arranged so that only rarely accessed things are in the same area as the lock. The chance of 2 or more CPUs trying to read/write to any of the items in the same cache line as the lock (including the lock itself) is very small. This could have been improved (every lock in it's own 128 byte area), but the extra memory cost wasn't worth the negligable gain.

>>>>Also, a micro-kernel is often small enough that the equivelent of a "big kernel
>>>>lock" is practical, meaning you could have one lock in the entire system in addition
>>>>to any "processes specific" locks, and no other locks.
>>>
>>>I don't know what systems that thing would run on, but it
>>>wouldn't be any big ones, that's for sure.
>>
>>That depends what the OS is designed for and how fast the slowest kernel function is.
>
>The microkernel handles scheduler, interrupts, IPC, and
>queueing messages to servers? And it has a big kernel lock?
>Then scalability sucks, full stop.

If the OS is designed for UP and SMP with 4 or less CPUs, and if the slowest kernel function is relatively fast, then it may be worth sacrificing a tiny amount of performance for simplified locking.

Given that (IIRC) Linux itself used a "big kernel lock" and nothing else for version 2.0, and that this lock is still used (although for a lot less than it originally was), and that the slowest kernel function within Linux 2.0 would've been much longer, I don't think you're in a position to argue too much.

It's just one of many compromises - scalability vs. developer time in this case.

>>>>Another potential advantage is NUMA systems, where you can set the CPU affinity
>>>>on a device driver so that it's always run on a CPU that is "close" to the I/O controller
>>>>used by the device. This doesn't apply to things like
>>>
>>>No, this is not. Because if the data is coming from / going
>>>to somewhere off node, you must send it across nodes anyway.
>>>And now you *also* have control information (messages) going
>>>across nodes as well.
>>
>>For control information you've got a choice - access the control data across nodes,
>>or access the device driver code and it's local state across nodes. Unless everything
>>(control data, code and state) is on the same node it can't be "perfect" regardless
>>of what you do (for all kernel designs).
>
>That isn't a choice, that is a restriction!

It's a hardware restriction that can't be avoided, where software developers can choose a way to attempt to minimize the "inter-node" access penalties. I will assume that for Linux, you end up with penalties when accessing device driver code and the device driver's local state (and no penalties when accessing the control data).


Cheers,

Brendan