Freddie (freddie.delete@this.witherden.org) on February 1, 2023 8:54 am wrote:
> Also, I'll note that with static scheduling it is not necessarily true that P cores will operate at the performance level of E cores.
> So long as OMP_PROC_BIND (and related variables) are not set, the OS scheduler is free to move threads around. Hence,
> when the P cores finish (and go to sleep) the scheduler can shift a task over from the E cores to them.
>
> Contrived example. We have 32 work items on a system with 4P and 4E cores with no SMT. Lets assume a P core can do
> 2 items a second and an E core can do 1 item a second. With a static schedule each core gets 32 / 8 = 4 items.
>
> After t = 2 seconds the P cores are done, and the E cores have 2 items remaining. Scheduler sees this, and shifts the E core
> threads to the P cores. At t = 3, we're done (maybe sooner as the E cores being idle may allow the P cores to boost higher).
> In contrast, a system with 8E cores would take until t = 4 to finish.

Thank you for your interesting educational example. You’re right about static scheduling not being as bad as I said because the OS scheduler can move threads from E cores to P cores. Suppose there were 48 work items in your example. With a static schedule, each core would be assigned 48/8 = 6 work items. The P cores would finish at t=3 and each E core would have 3 work items left to do at that point. The OS scheduler would move the E core threads to the P cores and each P core would complete those 3 items at t=4.5.

Now suppose the programmer used OpenMP’s auto schedule policy and the OpenMP runtime was smart enough to notice, after the E cores complete one work item, that the E cores are taking twice as long as the P cores per work item. Since the auto schedule policy allows the OpenMP runtime to figure out the best schedule, the runtime could, in theory, assign 8 work items to each P core and 4 work items to each E core. In this case, the loop would complete at t=4 instead of t=4.5. Does it seem practical for an OpenMP runtime to do this when the auto schedule policy is used? Is there any way for a programmer to manually give the P cores twice as many iterations as the E cores so the loop completes at t=4?

Regarding Heikki Kultala’s comment: Apple’s hardware does not have SMT so splitting a physical P core into two virtual P cores is not possible as a way of making the performance of the all threads in the system more uniform.