Ricardo B (ricardo.b.delete@this.xxxxx.xx) on January 22, 2014 6:58 pm wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 21, 2014 8:44 pm wrote:
> >
> > On the contrary. "The Filter Cache: An Energy Efficient Memory Structure" (Johnson Kin et al., 1997)
> > examines caches of up to 256 bytes and still concludes them to save a considerable amount of power,
> > while line buffers are typically just a single cache line. So if the filter cache is power efficient,
> > why not the symbolic L0 cache which doesn't even require addresses to be computed?
>
> Hit rate, hit rate, hit rate.
> Those papers are evaluating tiny caches for scalar workloads, where
> they actually get good enough hit rates to save power and time.

Yes, scalar accesses increase the hit rate. However, instructing the compiler to reuse memory operands at liberty, as observed from the Intel compiler, also increases hit rate. And those are the hits we care about for the L0 cache I'm proposing as an explanation for the KNL code we're observing.

> Considering the most favorable case, if you have a workload that consists on sequential access
> of an array of 8 bit elements, you'll serve 32 memory operations from a single 32B cache line.

You can't assume the best case. A filter cache has to be worth it for typical and even less favorable workloads as well. So it becomes a fine balance. That's not the case for the L0 cache because these are not the data reuses we're looking for.

> On the other extreme, if you vectorize the workload with 256-bit SIMD,
> you will only need a single memory access per 32B cache line.
> So, you get no re-use of your cache line from there.
>
> The fundamental issue with your proposal is that is too small for the
> workload in question: a 4x64B cache only holds 4x512-bit elements.

Merely two entries are required for the code examples that have been posted, to only require one L1 read port. A 2-entry L0 cache has a high enough hit rate for all the other loads. This is the essential difference between the filter cache and my proposed L0 cache: the filter cache's goal is to reduce the 'natural' number of accesses to L1 to save power, which creates a fine balance between size/power and hit rate, whereas my L0 cache's goal is to provide more read ports to allow more instructions to have a memory operand without blocking.

Similar to GPUs, typical KNL workloads would theoretically be fine with a 2:1 ratio of FMA units to load units. However, this makes the compiler's task more difficult by increasing register pressure and requiring delicate scheduling, while also putting a big demand on the register file. The L0 cache would kill several birds with one stone: you can freely reuse memory operands, the scheduler only has to worry about not previously accessed data, and the register file doesn't have to provide as many operands per cycle. And contrary to the tightrope you have to walk with a filter cache, you achieve the desired hit rate with a mere two entries.

So it's absolutely tiny and power efficient.