Brendan (btrotter@gmail.com) on 5/15/06 wrote:
>
>I _do_ understand this, I just don't see how seperating
>everything so that only one process has access to each
>"thing" would make scalability suck or make me 10 years
>behind.

Seperating some things is a wonderful thing, and not ten
years behind. Almost all lockless algorithms depend on
per-cpu data structures, and that's fine.

The problem is doing it for "everything". It doesn't work.

Doing it for the filesystem caches, for example, is a total
disaster. It increases memory pressure by huge amounts, and
makes synchronization much harder.

The synchronization issue isn't necessarily a huge deal
for regular file data (writes are much less common than
reads, and synchronization rules are fairly lax), but that
is not true for most filesystem metadata, for example.

In particular, under many loads, the bulk of the cached
filesystem data ends up being not the file data itself, but
the directory and inode information. And that also tends
to be the hottest part.

You may have thought that "read()" was performance
sensitive? Nope. Doing a "stat()" is under many loads the
much more performance-sensitive operation, and there the
name and inode lookup is the biggest deal, and there the
synchronization requirements are also a lot more strict
(ie you simply cannot allow two processes to create
the same filename. That would be instant bad OS karma).

So replication simply isn't a generally acceptable model.
It works fine sometimes, and when it works, it's
wonderful, and monolithic kernels will certainly do it too.
It's just not a "generic" solution.

In many other cases, replication is absolutely the last
thing you'd ever want to do, either because it doesn't
buy you anything (read-only data caches wonderfully), or
because the modification costs go up way too much. If you
write to it even occasionally, the cost of spreading out
the writes can be absolutely prohibitive.

>Cacheline contention caused by the message queue lock in
>a micro-kernel will be as bad as cachline contention
>caused by a device driver's own lock in a system where
>the device driver runs in the context of some other
>process, so swapping one for the other doesn't seem to
>make any difference to me.

Generally, you want very hard to run all queuing on the
same CPU that the service end result is needed on. The
lock ends up fairly expensive (and forget about contention:
contention doesn't really exist much in real loads for
good locking).

The expense comes from just the CPU having to serialize
enough for the locking to actually be effective. But at
least you can try to avoid having the cacheline move away,
either for the lock, or for the actual data being modified
(notably the pointers for the queues).

That said, you'll get cache bouncing at any time you have
multiple processes touching the same thing, that just isn't
avoidable. The best you can try to do is to have CPU
affinity, and make sure that as much as possible of the
setup is run in the same context (ie if you absolutely have
to have a "service thread", make sure that it is run on the
same CPU, and preferably with the same TLB as the process
it serves).

>If anything, a system where the device driver runs in the
>context of some other process would have worse cacheline
>contention and worse scalability because of the
>device driver accessing it's own local data from any CPU
>that happened to run it.

What local data?

Last I saw, there's basically no local data except for
the data that needs to be shared anyway (ie, the queue),
and the data you end up having to work with (which is
coming from the CPU that happened to run it).

Linus