Hi,

Linus Torvalds (torvalds@osdl.org) on 5/9/06 wrote:
---------------------------
>Brendan (btrotter@gmail.com) on 5/9/06 wrote:
>>myself (-@-.com) on 5/9/06 wrote:
>>>
>>>You don't get freed from those things in a microkernel
>>>either - those disadvantages are inherent to both models.
>>>Nobody said that a monolithic kernel fixes everything
>>>magically.
>>
>>If a device driver is running as a single-threaded process,
>>why would it need any re-entrancy locking at all?
>
>Why do you think that has anything to do with microkernels?

Because in my experience monolithic kernel designers dismiss the possibility, while micro-kernel designers prefer to implement device drivers as seperate processes. You are correct of course - it's entirely possible to implement device drivers in a monolithic system in a similar manner...

>Monolithic kernels have threads. It would be entirely
>possible - although pretty stupid - to do a single driver
>thread on a monolithic kernel, and have a simply queueing
>model for delivering interrupts and requests to that driver.
>
>Now, basically nobody would be as stupid as to do
>this on a monolithic kernel. It's a pretty broken model,
>and since the only reason to do it is to do something that
>is actually pretty simple anyway (locking is very simple
>if you just want to have total mutual exclusion), there's
>just no real upside.

Total mutual exclusion is easy until an IRQ handler needs to retrieve or store data without causing potential deadlocks (i.e. IRQ handlers would either need to be lock-free or _extremely_ careful).

For the upsides, if the system is designed so that only one process has access to each "thing" (memory buffer, piece of state information, etc), then all locks are either confined within the micro-kernel itself or process specific (in the case of multi-threaded processes), which reduces complexity as the majority of re-entrancy concerns for the hundreds of device drivers are handled by the kernel's IPC mechanism. This basically means device driver programmers either don't need to care about re-entracy at all, or they need to do the same things that anyone working on multi-threaded user level code does (if a device driver is implemented as a multi-threaded process). Also, a micro-kernel is often small enough that the equivelent of a "big kernel lock" is practical, meaning you could have one lock in the entire system in addition to any "processes specific" locks, and no other locks.

Another potential advantage is NUMA systems, where you can set the CPU affinity on a device driver so that it's always run on a CPU that is "close" to the I/O controller used by the device. This doesn't apply to things like network device drivers very much (as message data and/or shared memory can't easily be tied to a specific NUMA domain) but would benefit some things - e.g. file system code and the disk device driver both running on the correct NUMA node. Of course Linux can probably do this with kernel threads too.

Then there's I/O port protection, where the kernel could be designed to grant a process access to specific I/O ports rather than all I/O ports (which can't be done on 80x86 when device drivers run at CPL=0). This is the only way I can think of to prevent security holes in device drivers from being used to mess with the chipset, PCI configuration space, etc. On an unrelated point, there's probably no reason why Linux couldn't do this for "device drivers" that already run as user-processes, like the X server (for one scenario, see http://www.securityfocus.com/columnists/402).

Then there's hardware changes. For example, consider a server with hot-plug PCI. It runs for 5 years with no problems, then someone wants to plug in a new PCI device to handle increased load. The hardware is designed to handle this without downtime and the kernel supports hot-plug PCI so you'd think it could be done without downtime. Unfortunately the kernel is 5 years old and the device is new so you need to have downtime to update the kernel because of the new device driver. Someone's bound to say that Linux supports dynamically loaded modules, and in theory they'd be right (but in practice the kernel would've changed too much in 5 years to allow that possibility).

Lastly there's booting. If device drviers are seperate processes, then the "device detection" code (PCI bus scan, etc) can start the processes/device drivers as it finds them and continue, such that all device drivers are initialized in parallel. This could be extended to system services - for example, as soon as a storage device driver completes it's initialization suitable file system code is loaded, as soon as the video card is initialized a "virtual terminal" is started, etc. Doing device detection, device driver initialization, system service startup, file system checks (and possibly X server and GUI startup) all in parallel would slash boot times noticeably. There's already people trying to do "parallel system service startup" - I even saw one web page saying they reduced a 3 minute boot time down to 100 seconds using this method.

It'd probably be possible for Linux (or any monolithic system) to do all of the above, just like you can probably make a truck drive on water if you add enough "work arounds". It's not quite the same as something originally designed for the purpose though (like a boat), and you'd have to wonder at what point the monolithic kernel ceases to be monolithic.

IMHO the whole "micro-kernel vs. monolithic" thing (and OS design in general) comes down to compromises - balancing performance, features and development time to achieve a desired end result. For different end results, different compromises are made along the way.

>The reason microkernels often have that model is that if
>you don't have direct interrupt delivery, and if all the
>events are from some message queue anyway, it's just a
>very natural way to do things. You actually tend to need
>to do extra work to get re-entrant services and interrupts
>delivered directly.
>
>Of course, if you want good performance, heaven
>forbid, you actually want to get interrupts delivered
>directly, and you want to actually look at the whole
>message queue, not just the top entry, and suddenly you
>find that microkernels are actually in your way of
>doing things well (or that you actually need to know
>about things like locking, because you need to look into
>the message queue by hand).

My prefered method is to have a scheduler that gives "interrupt handling" threads a higher priority than all other threads, such that their message queues are (under normal circumstances) empty anyway. A message sent from a lower priority thread to one of these high priority threads causes an immediate thread switch followed by message/IRQ delivery. More complex device drivers would be multi-threaded, consisting of a very high priority thread for dealing with the hardware and handling IRQs, and another thread for dealing with "client requests", etc. The price (as compared to a traditional monolithic OS) is extra interrupt latency caused by the context switches (but there isn't a need to scan the message queue in this case).

>In general, locking isn't the problem in drivers, the hw
>is. Undocumented, buggy, and with strange interactions
>("Oh, DMA doesn't work properly with that other chipset?").
>
>(Admittedly, equally often the driver problem is all the
>infrastructure, and the fact that you have about a zillion
>drivers that use old and broken rules that it turns out
>doesn't work really well when you have a thousand disks
>directly connected. Oh, well).

Unfortunately those problems don't go away regardless of what you do (unless you limit yourself to a well-defined architecture, like the Commodore64)... :-)


Cheers,

Brendan