Hi,

Linus Torvalds (torvalds.delete@this.linux-foundation.org) on May 21, 2022 12:06 am wrote:
> Brendan (btrotter.delete@this.gmail.com) on May 20, 2022 8:42 pm wrote:
> >
> > Just because a thread got migrated to a P core doesn't mean it has to stay there - you could migrate
> > a thread back to the E core for a while (until it uses the library again) if you want.
>
> That's the "hey, this can be fixed" thing.
>
> But the unfixable thing is much more fundamental: 'cpuid' is suddenly not reliable or meaningful.
>
> Basically, what does 'cpuid' mean?

If you dig into CPU errata (the risk of CPUID misreporting), and the differences between different vendors, and the differences between the same vendor over time (e.g. about 5 different methods to obtain something as simple as L2 data cache size going back all the way to "use vendor/family/stepping and lookup tables"); it's easy to argue that CPUID has been broken for 20+ years; and this is made worse by more CPU errata (e.g. "TSX doesn't work properly, but it's reported as supported").

Ideally a good OS would examine CPUID and auto-correct everything (and auto-enable work-arounds for things like "accessed/dirty flags don't work", etc), and put all the info into a sane "same for all CPUs regardless of vendor" format; and then everything in user-space would be dissuaded/banned from using raw CPUID (and only allowed to use the sanitized/corrected/standardized info from the kernel).

In this case; if a thread is restricted to one CPU type it can obtain information for that CPU type; and if a thread is able to be migrated between 2 or more CPU types the kernel can provide information that is compatible with 2 or more CPU types (e.g. AND feature flags, report cache characteristics as "minimum only" or both "minimum and maximum", etc).

Of course "similar but worse/less flexible" can be achieved if CPUID causes a trap (the current solution for hyper-visors that support migrating VMs to completely different computers).

> Does it mean "what are the capabilities of the CPU I happen to be running on right now"? But
> then it's useless in any system where the load can be migrated to another CPU at any time.
>
> Or does 'cpuid' mean "Ok, I now give you a set of capabilities that I guarantee"? So now any
> process that has ever run 'cpuid' will be tied to a code that matches what it was told?
>
> In other words, you are looking for an engineering solution to "oh, this core doesn't do instruction
> XYZ", but you are missing the much more fundamental issue. Intel by design has very much exposed
> that whole "query what the CPU supports" thing as a native instruction, and you cannot make
> that instruction work with sane semantics in a heterogeneous system.
>
> And 'cpuid' is not some small implementation detail. It's literally what any core system
> library would use to decide "How do I choose implement functionality XYZ?". So 'cpuid' has
> to work, and it has or be reliable and meaningful, because it's literally how people will
> make the decision on whether to use the AVX512 version of the library or not.
>
> If you claim you do AVX512, then all processes end up getting pinned to
> big cores just because some random library goes "oh, then I'll use it".
>
> And if you claim not to do AVX512, then people won't be using
> it at all, and you would be better off not having it.,
>
> And if you randomly return a value based on "right now you happen to be running on CPU X, so you do or do not
> have AVX512 based on that", you end up with random performance and the worst of both of the above worlds.
>
> And that is all assuming that the system software bent over backwards to make the whole thing work
> with auto-migration in the first place, so all these bad outcomes actually require a fair amount of
> engineering to even work at all (ok, except for the "never report AVX512" case, of course).
>
> End result: you can't win.

How about we create an OS where unsupported instructions are emulated; so that you can run a new executable that uses AVX-512 on a crusty old Prescott Pentium 4 from 2005 without any problem (other than performance); and where a "some cores don't support AVX-512" problem becomes a performance issue (a minor addition to the "P cores are faster than E cores" performance issue you already have to deal with)?

> So you're answering the wrong question entirely. The question was never "Can I auto-migrate
> a process that uses AVX512 to a big core that supports it, and maybe auto-demote it later?"
>
> No. The question was much more fundamental: 'what does cpuid report?'.
>
> And that question simply has no valid useful answer in the heterogeneous system.
>
> Ergo: the heterogeneous model is broken. Fundamentally and unfixably so.

Nonsense. E.g. Intel could easily say "old CPUID leaves (with bit 15 of EAX clear) report information that's compatible with all cores; but (for CPUs that have mixed cores) here's a whole new set of CPUID leaves (with bit 15 of EAX set) that report information that only applies to this specific CPU" so that old software that checks CPUID works and new software that supports different CPUs can work better.

- Brendan

> (And as always: in embedded systems you can do anything you want, since you control the horizontal
> and you control the vertical. And so you can just keep big and small cores separate and never
> migrate things at all, or only do it for loads that explicitly have asked for it)
>
> Linus