NoSpammer (no.delete@this.spam.com) on January 19, 2014 1:48 pm wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 16, 2014 3:17 pm wrote:
> > The 40% is for the same arithmetic instruction, with a memory source operand instead of a register,
> > hitting L1. So accessing L1 has a very substantial cost of 40% per cycle, not 13.33%.
>
> That's bad then but that's why there are registers.

Sure, but we can do even better than that. The L0 cache would only have a few entries and two read ports, versus the register file which has 32 entries and multiple read ports. So accessing the L0 as often as possible instead of the RF, saves power, and on top of that the RF's read port count can be reduced for even more savings. Not only that, reusing data from memory by using a register requires adding a separate load instruction. Finally, the L0 cache would allow to have just one L1 read port instead of two.

So the L0 could achieve no less than a quadruple power saving, while with only a couple of entries it would cost next to nothing itself.

> > It's not a fixed idea. I can let go of it in a heartbeat. But someone still has to
> > provide any reasonable explanation why an extremely tiny cache which only requires
> > a handful of small comparators to find hits on reads and invalidate entries on write,
> > would not be power efficient enough to save some of the L1's 40% access power.
>
> No one disputes this.

Actually several people seem to disagree, but I'm glad you think it's undisputed.

> The question is about the utility especially in the power hungry cases
> where L1,L2 & memory controller are fully exercised but L0 is only burning some extra power.

GPU manufacturers are putting very considerable effort into optimizing the power consumption of the register files with smaller operand register files or register caches. So that should give you some idea of the importance of optimizing the accesses to the data that is already the closest to the execution units. The things is that while an L2 access consumes more power than an L1 access, the latter in general occurs much more frequently. Likewise the L0 accesses would occur even more frequently since the compiler is directed to reuse memory operands instead of registers.

I'm sure there are a few scenario's where you're bottlenecked by L1 (or L2 or RAM) and the L0 is just burning (a tiny bit of) wasted power. But that's true for the L1 as well when you're bottlenecked by L2. Heck it's true of the register file. So you have to evaluate things over the entirety of use cases. Since the compiler is instructed to reuse memory operands, the L0 would typically be used very frequently and save a lot of accesses to L1 which would have costed substantially more power.

> > This isn't about a minor tweak in "optimality". Loading data from memory again when it's already
> > in a register is a big deal and yet a trivial optimization. They do it for other architectures.
> > So I don't think I'm being naive about having faith in Intel's compiler engineers being capable
> > of this optimization. They must have disabled it deliberately for a reason.
>
> I've been using Intel compilers for more than 10 years and I could fill at least 20 pages with stupidities
> generated by the compiler. It's work in progress. Sometimes a new version of the compiler generates worse
> code for the source at hand. Sometimes you get a better result targeting a different CPU. Looking at a few
> cases does not prove much. What I've also noticed with the latest versions is that you can get a rather
> generic compile of a function and a totally different compile when that same function is inlined in another
> loop. Sometimes a loop is not unrolled. Then you unroll it 4 times by hand and the compiler finally decides
> it will unroll it one more time. It's obviously a bit of a gamble which heuristic kicks in.

I'm sorry but I still don't think you appreciate just how radically different this is from typical compiler flakiness. All the examples consistently reuse memory operands, even in situations where it would be utterly trivial to use a register instead. These aren't corner cases where you might expect some hickups; it's very typical and straightforward code and it is downright impossible that the Intel engineers would not have noticed this. The AVX2 code looks entirely as expected so clearly they know what they're doing.