Ricardo B (ricardo.b.delete@this.xxxxx.xx) on January 9, 2014 5:21 am wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 9, 2014 3:07 am wrote:
>
> > > First, register files are lower power than caches...
> >
> > That is not universally true. Caches have fewer transistors per bit can operate at a longer latency,
> > so a sufficiently small cache is more power efficient than a register file. Also, an L1 access requires
> > additional power for the address generation/translation, but the L0 could use symbolic indexing.
>
> Nobody debates a 1-2 port SRAM array (ie, like the one used in cache)
> uses less power per bit than a N-port SRAM array (ie, register file).
>
> What you aren't showing is that you can implement all the surrounding logic (CAM logic, invalidation logic)
> and use less power than the straightforward option: load the value in a register and reload it.
>
> I think you fail to appreciate cache isn't just a SRAM array. There's a complex, power hungry
> and relatively slow amount of logic to implement the CAM, replacement, invalidation, etc.

I do realize there's more to a cache than its SRAM array. But when you say those things are complex, power hungry and relatively slow, I think you're prejudiced by their cost for a typical cache. The L0 cache I'm suggesting as a possibility is very different (for starters it's read only), and should perhaps not even be called a cache to avoid confusion. Line buffers and store queues are more similar to this L0 cache than the L1 cache.

The CAM we're talking about here is extremely small. You need two 20-bit comparisons per cycle to look for a match in the L0 cache. That's it. Ridiculously cheap. Replacement and invalidation would be similar in complexity to a line buffer and store queue, i.e. things the LSU already does anyway. Note that a second L1 read port also comes at a cost beside the second SRAM lookup and address generation.

So I think you're making a big deal out of nothing, and the extremely tiny SRAM size versus fairly large register set size would be the dominant factor in achieving a net power consumption reduction.

> > KNC consumes only 40% more power for an instruction which acceses L1, versus an instruction which only
> > uses registers: http://www.eecs.harvard.edu/~shao/papers/shao2013-islped.pdf Thus it's easy to see that
> > an L0 cache with only a few entries would be more power efficient than both the L1 cache and the register
> > file. Keeping a single L1 load port for KNL and even reducing the power consumption over register accesses
> > with an L0 cache would be a very significant win Intel is not likely to overlook.
>
> That study does not break down the cost between the register/cache access and the op itself.

As the paper mentions, the op itself isn't a significant factor in the power consumption per instruction.

> > > Second, virtually addressed caches are a stupid idea. This should be evident from the fact that almost
> > > all high performance CPU caches are VIPT or PIPT. Also, how will this interact with threads?
> >
> > The L0 cache I'm proposing would not be virtually addressed. It would use the instruction's
> > encoding for the memory operand (i.e. the SIB byte, its EVEX extension bits, and part of the
> > displacement bits). So you're 'symbolically' matching a memory operand that has been loaded
> > before (e.g. [rax+r14*4] in Eric's example code). The entry would be invalidated if rax or r14
> > is written to, or when the entry's corresponding L1 cache line is modified/invalidated.
>
> It's still a form of virtual addressing.

There is no address being computed. But I can agree on calling it a form of virtual indexing (VI) if you want. But since VIPT caches are high performance, it's really about the tagging, not the indexing. What you and David appear to be confused about is that this isn't a virtually tagged cache. There are no tags. The L0 would not contain any data that isn't in L1. The symbolic encoding of the address is only used for lookup, not for validation. Any change to the L1 cache line corresponding to an L0 entry would immediately invalidate it.

> > > Third, as Michael S pointed out - this is insanely brittle.
> >
> > No more brittle than anything else happening in the L1 cache and LSU. It's basically a symbolically
> > indexed line buffer with multiple entries and two read ports (http://caps.cs.binghamton.edu/papers/ghose_islped_1999.pdf).
> >
> > Shouldn't be too much of a challenge for Intel to design something like this.
>
> It's brittle in the sense that with only two entries, hit rates and thrashing would be terrible.
>
> Which brings me to another point you don't appreciate: thrashing is activity and activity is power.
> A cache that does not get decent hit rates wastes power, performing
> lookups which fail and replacing entries it will never serve.
> And your proposal (2 entries) is too small to achieve decent hit rates.
> Outside specific code sequences, the vast majority of loads will not be served from it.

First of all there is no thrashing in the classic sense. It's read-only. You can perform many writes without invalidating the L0 entries. They only get invalidated when the corresponding L1 line gets changed, or the GPR registers it uses are changed. There is only writing activity when reading from L1.

Secondly, it doesn't require a high hit rate for a tiny cache to be of value. A line buffer is doing a good job if it achieves a 50% hit rate. Any hit saves power, and a miss isn't a big deal. Likewise the proposed L0 cache's main job would be to provide a second source memory operand and thus increase performance, but as an added bonus every hit also lowers the power consumption over requiring an L1 access or a register file access. Just a couple of entries suffices to have a good enough hit rate to achieve those goals.

> > > Fourth, the coherency implications are not considered.
> >
> > That's because there are no coherency implications worth mentioning. It can be kept coherent with
> > L1 at all time. Everything the LSU already does to keep L1 coherent, it can apply to L0 as well.
>
> You still need logic to associate the L0 symbolic (virtual) tags with the L1 entries
> and ensure the L0 entries get invalidated when the L1 cache line gets invalidated.
> It does not happen by magic.

I know. But it's negligible. A 32 kB L1 has 512 cache lines, so you only need 9 bits for each L0 entry, and a valid bit. So together with the 20 bit symbolic address, that's 30 bits of bookkeeping data for every 512 bits of payload. On a hit it saves you from doing an L1 lookup involving address generation, a TLB lookup, a tag lookup and match (25 bit), etc. Invalidating an L0 entry on a L1 write merely involves checking the 9 bit index and clearing the valid bit when they match. Sure, it's not magic, but are you seriously arguing about a 9 bit compare? For a second L1 read port you'd need a 25 bit compare, after looking it up.

> > > Fifth, why would you want to create ANOTHER structure that stores must check?
> >
> > The L0 cache would be read-only.
>
> But your L0 needs to be coherent with the stores.

Which only requires clearing the valid bit after a 9 bit compare. It's a non-issue.