Wes Felter (wmf.delete@this.felter.org) on June 16, 2022 4:57 pm wrote:
> Anon (lkasdfj.delete@this.fjdksalf.com) on June 16, 2022 3:06 pm wrote:
>
> > For those scratching your heads, this dude's an armchair quarterback who believes the main reason
> > Apple's chip architects put in many hardware codec blocks was to accelerate single stream encode/decode
> > to a higher degree than a single block could on its own. He just *knows* it must be possible to
> > split the work into multiple threads so that multiple codec "cores" can collaborate on it - some
> > software codecs are multithreaded so obviously that must be possible with any hardware codec design
> > too. Therefore, according to him, if Apple's system can't do that, it must be a terrible and embarrassing
> > failure of communication between hardware and software departments.
>
> It is possible to split video on GOP boundaries and encode in parallel,
> even over a cluster. For example, here's what Netflix does:
>
> "The media content is broken up into smaller chunks. Each of the chunk is a portion of the
> video, usually about 30 seconds to a couple of minutes in duration. In a massively parallel
> way, we encode these video chunks independently on our servers. Once all the chunks have finished
> the encoding process, they're reassembled to become a single encoded video asset."
>
> Maybe this can't be hidden under the existing VideoToolbox API so it probably requires app
> changes. And maybe those changes haven't been made which (outside of Final Cut) isn't Apple's
> fault. But I can understand why customers might be confused or frustrated if they are waiting
> on a single-stream encode workload which isn't faster on higher-end Macs.

Even something apparently as simple and basic as video encoding remains under substantial construction because Apple are doing so much that is novel.
For example a substantial part of video encoding (certainly, even still, for all the MPEG codecs; I would guess so also for ProRes) is knowing the optical flow between frames, something that has traditionally been expensive to compute. If you know optical flow reliably, you can in turn use this to drive many other traditionally expensive decisions like how to break up the tree of blocks to sub-blocks in a frame.
Now, something that was clear if you viewed the the right WWDC videos, is that Apple have a new Optical Flow net (actually I believe it's version 3 of this particular net) that's substantially better than what went before.

Point is, to simply claim "the video can be broken up via scheme X because that's what company Y does" is not helpful. It may be true that the video can be broken up that way, but there may also be very good reasons why Apple is doing things differently.
For example the Netflix scheme you describe may require vast amounts of physical DRAM, to an extent that is acceptable to Netflix (and to Apple for their internal encoding) but is considered not acceptable, at least right now, for the bulk of their users.

Alternatively it may simply be that Apple are working through these things one step at a time based on a schedule of what they believe is most important (probably informed by customer discussion) rather than what the internet believes is important.
If very interesting things can be done in terms of encoding by use of the NPU, that is clearly worth pursuing, but it also may mean that other interesting things worth doing (like making full use of multiple media encode engines in all circumstances) may have to wait a year or two. Apple aren't *just* trying to figure out all the ways to accelerate ProRes; they're also getting ready for VVC; they're also constantly working on the quality front (issues of HDR and higher frame rates); they're also working on the capture side (especially capture quality so as to help amateurs); they're also working on the networking side (eg faster random access).

It's fine to say that "they should do better"but I don't see anyone else running any faster in this space. Handbrake, for example, is a great app I use frequently, but I don't see them exploring how NPU (or even GPU) could be used to improve encoding; they're great at endlessly refining something that's already been done, not so great at exploring completely new ways of doing things.
Things just take time, especially when there's a lot of original work going on.

In other contexts (like GPU) one sees the same thing. Concepts were added to Metal two or three years ago; and it is only this year that we see higher-level APIs now exploiting those additions so as to make them available to non-specialists.