Brendan (btrotter@gmail.com) on 5/13/06 wrote:
---------------------------
>Hi,
>
>nick (anon@anon.com) on 5/13/06 wrote:

>>No, it would suck. You're about 10 years behind: lock
>>contention doesn't hurt scalability any more, cacheline
>>contention does.
>
>Am I?
>
>For my last prototype, each thread has a "thread control block" containing it's
>message queue lock (one queue per thread). While each lock wasn't alone in it's
>own 128 byte area (Intel's recommendation) the data around

That's because it is the size of their largest dcache line.

it was deliberately arranged
>so that only rarely accessed things are in the same area as the lock. The chance
>of 2 or more CPUs trying to read/write to any of the items in the same cache line
>as the lock (including the lock itself) is very small.

Doesn't matter, if they have to both take the lock ->
cacheline bouncing; if one CPU writes a messages which is
read by another -> cacheline bouncing.

>>The microkernel handles scheduler, interrupts, IPC, and
>>queueing messages to servers? And it has a big kernel lock?
>>Then scalability sucks, full stop.
>
>If the OS is designed for UP and SMP with 4 or less CPUs, and if the slowest kernel
>function is relatively fast, then it may be worth sacrificing a tiny amount of performance for simplified locking.

Even at 4 CPUs, it is not a tiny amount of performance!
(unless you're talking about 4 cores sharing cache, which
you're not).

On some workloads (eg. pagecache or scheduler intensive,
because each of those subsystems had a single lock), Linux
2.4 would barely get 2x the throughput of a single CPU on
a 4 CPU system (and 2.4 had hundreds/thousands of locks).
I could probably dig up some links if you're interested.

>
>Given that (IIRC) Linux itself used a "big kernel lock" and nothing else for version
>2.0, and that this lock is still used (although for a lot less than it originally
>was), and that the slowest kernel function within Linux 2.0 would've been much longer,
>I don't think you're in a position to argue too much.

Err, I don't follow your logic at all. Linux 2.0 has a
global lock, scalability sucked. What is my position, and
why isn't it one to argue too much? How much am I allowed
to argue?

I think your position of arguing unproven/unimplemented/
untested ideas is much more tenuous than mine.

>
>It's just one of many compromises - scalability vs. developer time in this case.
>

Exactly. Don't try to pretend it would be scalable though.

>>>For control information you've got a choice - access the control data across nodes,
>>>or access the device driver code and it's local state across nodes. Unless everything
>>>(control data, code and state) is on the same node it can't be "perfect" regardless
>>>of what you do (for all kernel designs).
>>
>>That isn't a choice, that is a restriction!
>
>It's a hardware restriction that can't be avoided, where software developers can
>choose a way to attempt to minimize the "inter-node" access penalties. I will assume
>that for Linux, you end up with penalties when accessing device driver code and
>the device driver's local state (and no penalties when accessing the control data).
>

In Linux, the control data does not need to go across
node because it is basically kept in the stack of the
running process. Instruction text is a different matter,
but AFAIK, that can be replicated on the big NUMA machines
in Linux.