dmcq (dmcq.delete@this.fano.co.uk) on July 26, 2021 5:38 am wrote:
> Michael S (already5chosen.delete@this.yahoo.com) on July 26, 2021 2:53 am wrote:
> > Adrian (a.delete@this.acm.org) on July 25, 2021 9:16 pm wrote:
> > > dmcq (dmcq.delete@this.fano.co.uk) on July 25, 2021 5:36 pm wrote:
> > > > Introducing the Scalable Matrix Extension for the Armv9-A Architecture
> > > >
> > > > I noticed ARM have issued yet another of their planned future
> > > > architecture blogs, this one is called "Scalable
> > > > Matrix Extension". Same sort of idea as what Intel is implementing as far as I can see. Except there is
> > > > a strange aspect in that it requires a new mode for SVE called "Streaming mode" SVE. They talk about having
> > > > the new SME instructons and a significant subset of the existing SVE2 instructions. And they say that
> > > > one could have a longer length for the registers in streaming and non-streaming mode. As far as I can
> > > > make out in fact only very straightforward operations are included in streaming mode.
> > > >
> > > > I guess that instead of being 'RISC' instructions these would have a loop dealing with widths greater
> > > > than the hardware SVE register or memory or cache width. Thy've implemented something like this in
> > > > the Cortex-M Helium extension, but I'd have thought they could just rely on OoO for the larger application
> > > > processor If it is so they can have larger tiles in their matrix multiplicatin I'd have thought there
> > > > would be other tricks that could do the job without a new mode. However I can't see they would have
> > > > put in a new mode without it being very important to them. Am I missing something?
> > >
> > >
> > >
> > > You are probably right about the necessity of looping in certain cases.
> > >
> > > They explain clearly enough why a streaming SVE mode is needed, to be able to present to
> > > software an apparent vector register width that is larger than the width of the ALU's.
> > >
> > > This "streaming" mode is actually exactly like the traditional vector computers have operated. For
> > > example a Cray-1 had an apparent vector register width of 1024 bits, but the vector operations were
> > > computed by a 64-bit pipelined ALU, in multiple clock cycles, i.e. "looping", like you say.
> > >
> > >
> > > They also explain clearly enough why extra matrix instructions are useful. The throughput
> > > of a computational program operating on large data structures on a modern CPU is normally
> > > limited by the memory throughput, usually by the memory load throughput.
> > >
> > > For many problems you can reach easily the memory throughput on any CPU and
> > > there is nothing that can be done to improve the performance above that.
> > >
> > > However the problems that can be solved using matrix-matrix operations are an exception, because for large
> > > matrices the ration between arithmetic operation and memory
> > > loads can become arbitrarily large, so increasing
> > > the number of arithmetic operations that can be done per memory load can increase the performance.
> > >
> > > The only way forward to greatly increase the number of arithmetic operations
> > > per memory load is to add matrix instructions,
> >
> > It depends on the definition of "greatly" and on specific operation. For the simplest, but fortunately
> > the most common, operation of matrix multiply, it is possible to achieve quite good ratio
> > with quite small modification to AVX512-like ISA.
> > All we need is ability to broadcast one of the multiplicands of SIMD FMA from
> > arbitrary lane in register vector. AVX512, in its current form, can do it
> > from memory, but not from register, which is good, but insufficient.
> > So, what become possible with such minor change in the ISA?
> > Let's assume double-precision matrix multiplication, ISA with 32x512-bit software-visible
> > vector registers and 2-4 times as many physical vector registers in OoO back end.
> > In the inner loop we multiply 10 rows by 16 columns (=2 SIMD columns).
> > 10x16=160 accumulators occupy 160/8 = 20 vector registers.
> > Additionally, we need 10 VRs for row inputs and 1 VR for column input (loaded
> > with 16 different values throughout single iteration of the inner loop).
> > So, overall we need 20+10+1=31 software-visible VRs, which is less than 32.
> > Each of 160 accumulators is updated 8 times per one iteration of
> > the loop, so there are 1280 FMAs per iteration == 160 SIMD FMAs.
> > On each iteration we load 10x8=80 row inputs + 8*16=128 column
> > inputs. Overall 208 double precision loads = 26 SIMD loads.
> > FMA-to-load ratio = 160/26=6.15.
> > Whether you call it "great" or just "large" is your call, but personally I don't expect that
> > any CPU core that is still considered "CPU" rather than "accelerator" would ever have FMA-to-load
> > ratio above 4, let alone, above 6. Right now, it seems, none of them exceeds 1.5.
> >
> >
> > > a.k.a. tensor instructions.
> >
> > If by "tensor instructions" you mean "outer product" (I think, nowadays
> > that's the most common meaning) than I tend to disagree.
> > IMHO, many tasks can be accelerated quite nicely by "inner product" instructions. And intuitively,
> > without being a chip architect, I would think that "inner product" is inherently more energy
> > efficient than "outer product", due to lower fan out to register files.
> >
> > But, as shown above, neither is needed for specific task of [big] matrix multiply.
> > Now, small or medium matrix multiply is completely different kettle of
> > fish. BTW, much more important for me, from practical perspective.
> >
> > >
> > > The instructions added by Intel in Sapphire Rapids have a very limited usefulness. Due to
> > > their low precision they can be used only for Machine Learning. The ARM extension appears
> > > to be much more general purpose, so it should be also useful for other applications.
> > >
> > > Unfortunately it is not clear how many years will pass until the
> > > introduction of ARM cores implementing this ISA extension.
> > >
> >
> > In case of Apple, the most probable answer is "never".
> >
> > In case of QC, it's still not clear if they are coming back to business of developing their
> > own cores. Although if they don't then why they paid astronomical sum fir NUVIA?
> >
> > In case of Arm Inc., 3 years sound too optimistic. 5 years - may be.
> >
> > As usual, it leaves Fujitsu. May be, they are already in process of doing it
> > and current announcement is just a posteriori documentation of their work?
>
> Apple already have a facility to do matrix multiplication operations. I don't know how it compares but
> it would be peculiar not to support these operations eventually especially in their desktop computers.
> And they would be involved in any discussions ARM have about future enhancements so they probably knew
> about this quite a while ago judging from how long ARM was discussing SVE before it described its plans.
>


Apple already added their own matrix instructions (AMX) with the A13 almost two years ago. They seem to be strongly encouraging developers to use library calls to access them, rather than developers writing their own code using them. The most likely reasons for doing that would be if they want to be able to make incompatible changes to AMX, or to be able to completely replace AMX down the road.

We can only guess whether Apple intends to continue going their own way with AMX, or it was intended as a placeholder until SME was ready. While Apple would have input into SME's design as one of ARM's biggest partners, they may prefer their own matrix instructions as they can be precisely targeted at Apple customers' needs. ARM needs to consider the whole market from mobile to HPC, Apple's product line isn't quite so broad.