Mark Roulo (nothanks.delete@this.xxx.com) on July 15, 2020 6:10 pm wrote:
> Romain Dolbeau (romain.delete@this.dolbeau.org) on July 14, 2020 11:42 pm wrote:
> > Fujitsu already moved to 256 bytes cache line in the A64FX, and I wonder what are the consequences
> > for non-HPC codes in practice. I'm not sure that's something that Intel would be willing
> > to risk in every markets for the sake of HPC, and I know I would need a lot of simulations
> > before I'd be comfortable with a major change like that from established habits. Unlike
> > changing vector width, changing cache line size affects everyone directly.
>
> The Fugaku supercomputer uses HBM2 memory so probably reads and writes memory in (at least) 2,048 bit chunks.
> If you are going to read and write 256 bytes at a time, you might as well make this your cache line size.
>
> You don’t have to do this. The Volta GPU also uses HBM memory and has cache lines of “only” 128 bytes
> (so the HBM can clearly be accessed in 1,024 bit chunks... but I suspect that the HBM had something to do
> with this decision. And IBM’s POWER9 uses DDR and has 128 byte cache lines, so there is that ...
>
>
>

A single HBM2 package has 8 channels that are accessed independently, each 128-bits wide, with either a burst length of 2 in legacy mode or burst length of 4 when split into two separate 64-bit pseudo-channels, so either way the basic transaction size is 32B.

http://monitorinsider.com/HBM.html#pseudo-channel-mode

By contrast, I think the native transaction size for DDR4 is effectively 64B. So no, this probably doesn't have anything to do with HBM2 per se, and is probably just a design choice based on the typical workloads Fujitsu expects to run. Wider cache line sizes means less spent on tags, so if you're fairly confident that you'll be filling up your dual 512-bit vector ALUs every cycle, you probably expect that you have enough spatial locality to make the tradeoff acceptable.