smeuletz (alexsmeu.delete@this.yahoo.com) on January 6, 2020 10:33 am wrote:
>
> Thus, kernel leadership should not be afraid to deprecate and remove obsolete things.

So this is something where we will fundamentally disagree.

The number one priority for Linux has been "don't break existing users". I have seen way too many projects that think that in order to make progress, you have to break things ("We're happy to announce version X, and you will now have to recompile everything to a new model to use it").

And I just don't believe in that "you need to break a few things to improve" model of development. We may change things internally, and break assumptions inside the kernel, but we do so without breaking old interfaces, even if we might find them very inconvenient.

In fact, one of the few rules that predate that one (and that "no regressions" rule goes back decades), was "make it easy to port existing Unix software". I came from a background where I wanted to write my my own kernel, but I had no interest at all in writing my own applications. That's still true, with a very few notable exceptions.

So Linux came from a background of trying to very actively follow POSIX (which includes that sched_yield()) in order to make those things easy to build under Linux.

So we don't remove support for old things and we don't dismiss basic standards lightly.

Both binaries and source code from decades ago are supposed to still "JustWork(tm)". They may not work optimally of course, but the kernel really does not try to force people to upgrade their existing setups. Quite the reverse: I want to make it easy for people to try new kernels exactly by pretty much giving a guarantee that if something regresses, we'll fix it.

So we'll blame ourselves for breaking something you used, not blame you for using it. We might be unhappy about it, of course, but it's our problem.

And that very much means that we continue to support old programs.

Now, that "guarantee" is not entirely absolute - it does depend on practical things like "oops, fundamental security issue" and also "you have to actually report the regression in a somewhat timely manner". But it's the closest thing to an absolute hard rule we have in the Linux kernel.

And yes, some people disagree about that rule, but it's a rule that Linux will continue to follow at least when I maintain it. Because I think that the one thing that DOS and Windows did right ware not about technology, but about backwards compatibility. People ran DOS and Windows not because they loved DOS and Windows, but because they fidn them useful thanks to all the programs available. Breaking those programs breaks the whole point of an OS.

I can't fix other projects that disagree with my approach, and I can't fix distributions that don't care, but for the project I maintain, I can hold to that rule.

And yes, despite that rule, we've occasionally removed things after years of warnings, but there needs to be serious maintenance reasons for that removal (and we try very hard indeed to make sure that it doesn't affect anybody at all - and we re-instate things if it turns out we were wrong). But that's not just "people can shoot themselves in the foot with this because it's a bad interface" kinds of things.

That said, the whole "make it easy to port existing Unix software" rule from way back when does mean that I'd love to also make it easy to port Windows software too. So don't get me wrong - if we can make that easier, we should do so. But at the same time, that doesn't mean that we should make bad technical decisions.

And honestly, it's usually not the kernel that holds the "port from Windows" side back. It's all the other infrastructure, including libraries, GUI stuff etc etc. So being compatible with Windows doesn't really tend to come up very much at a kernel level. I wouldn't object to it, but it just doesn't tend to be a big issue.

In this particular case, the fix really is "don't do spinlocks in user mode". That's a rule for other reasons too, and the expectations of "sched_yield()" behavior is only one small part of it. Linux broke developer expectations because Linux uses a per-cpu run-queue for scheduling (*).

And the thing is, a scheduler that uses per-cpu data structures is a good thing. But like all engineering decisions it's a trade-off, of course, and has downsides, and one of the downsides is "global behavior is more complex".

It does mean, for example, that "sched_yield()" is mostly about the "current CPU runqueue", and will largely ignore threads that are on the run-queues of other CPUs.

Using per-cpu run-queues makes the behavior of "sched_yield()" simpler and faster in many ways, but it also means that it becomes much more complex conceptually and you suddenly have those global issues - you're not yielding to something running on another CPU, you're basically yielding only to something that wants to run on this CPU. You really aren't moving the process to the end of one global list, you're only making a local decision.

The "use sched_yield for locking" mental model depends on one single run-queue for the whole system which the normal Linux scheduler simple doesn't do.

That mental model of using yield for locking is completely broken even in a single run-queue system, of course - because it also depends on running no other load at all. I really wasn't trying to be impolite when I said it is fundamentally broken. I was simply stating a fact.

Of course, if you come from a world where you often really are the only program running (and gaming often is that, particularly on consoles), you may be in for a nasty culture shock.

Not perhaps so different from the culture shock that people coming from DOS had when they had to care about things like permissions and other users etc. Sometimes it can be a big culture shock.

Linus

(*) Simplification. Linux has several schedulers, and they all have complex behavior, but as a rough approximation you can say "per-cpu runqueues is what caused the particular surprise with sched_yield()'.