Stubabe (Stubabe.delete@this.nospam.com) on January 17, 2014 12:58 pm wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 16, 2014 9:23 am wrote:
> >Haswell has four arithmetic execution ports instead of two, despite 'only' two load ports, >because
> they expect to have a purpose for them! Those two extra ports can also require a load >operand.
> If it was really necessary to have two load ports just for the two floating-point >execution ports,
> you couldn't execute many other instructions, let alone two! But Intel has >explicitly mentioned
> that a fourth execution unit was added to offload the vector ports. As a >consequence they also
> added a third AGU to sustain two loads per cycle, but nothing exceeding >that 1:2 ratio.
>
> You have heard of specfp_rate? Haswell's FMA units ARE constrained by its load bandwidth even now.
> With a latency of 5 you need 10 software visible registers just for the accumulators to fully exploit 2
> FMA units and you still have to source 2 other operands for each of those 10 instruction for registers/ram.
> Haswell with 2 load ports and 16 registers is already starved. That's ignoring the 2 other "spare" execution
> units. But those will typically be used to blend/shuffle or to do flow control/loop overhead/pointer respectively
> so no they don't go to waste just because there is insufficient load bandwidth.
>
> Knights [whatever] may have 32 registers but that not really enough to prevent high load bandwidth either
> as you still need at least one load per FMA for matrix stuff, and often have a high ratio of load:store so
> reusing the store port for loads is probably not a bad idea (I doubt they added a 3rd memory issue pipe).

One load per FMA is not necessary. NVIDIA's GK110 architecture can sustain its theoretical DPFLOPS rating in DGEMM, with two DPFMA units and just one load unit. It even achieves 45% of its peak performance in SGEMM with six SPFMA units and one load unit. So Haswell's two load units suffice to supply all four arithmetic ports with memory operands. You need Hyper-Threading to help hide the latency of Haswell's 5-cycle FMA, but Xeon Phi has a 4-cycle latency, twice the registers, and 4-way SMT. So one load port would suffice for KNL to sustain a peak IPC of 2 instructions.

> > > I'm confident there is no L0 cache. I never said that the L1D
> > > is dual ported, I said that it supports 2x64B reads/clock.
> >
> > Then it needs two read ports. Even if it's implemented with multiple banks with a single read
> > port each, it adds complexity to receive two addresses, select banks, and deal with conflicts,
> > and you're also consuming twice the SRAM lookup power on every dual access in the same cycle.
> > With a tiny L0 cache you could avoid all that and even reduce register file pressure.
>
> Yes

Great.

> but how it that implemented?
> Either in the Load units themselves -> Still need two Issue ports for Load + 2 writeback paths.
>
> OR
>
> Its an extra stage in the pipeline for floating point operations (and just uses the bypass network)
> I.e. Horrific on an OOO design as the scheduler needs to check the L0 indexes on each issue (as L0 hit/miss
> will effect instruction latency and coissue)or if we assume the FPU is in order (as in Silvermont) then that's
> an extra (stallable) pipeline stage that will introduce bubbles on a L0 miss - also power inefficent.

I assume the vector pipelines are in-order with a 4-cycle latency and round-robin scheduling, just like KNC. So the L0 cache does not affect scheduling. One L0 miss becomes an L1 access, so that's fine. Two L0 misses would create a bubble in one of the pipes, but that should be rare enough and cheap enough compared to the energy savings from L0 hits, and the savings from simplifying the register file.

> The real problem with a tiny L0 is its piss poor hit rate. A two entry structure would only work
> for A,A,A,A or A,B,A,C,A,D type load patterns, in any other case it will miss. So even if it does
> save power in a few corner cases its just an expensive (power) waste of die area in every other.

That's exactly the load patterns we're observing in the examples Eric has posted. Remember, the L0 hit ratio does not have to be very high. It just allows to reuse memory operands without requiring a second L1 load port and to allow reducing the register file complexity.

Here are some papers which clarify why a very high hit rate for an L0 like structure isn't necessary:
"The Filter Cache: An Energy Efficient Memory Structure" (Johnson Kin et al., 1997)
"Reducing Data Cache Energy Consumption via Cached Load/Store Queue" (Dan Nicolaescu et al., 2003)

> You appear only to consider the case where you L0 helps with out factoring the continued cost of it when
> it provides no benefit (during a miss).

First of all there are also continued savings from simplifying the register file. Secondly the L0 cache is utterly tiny so any cost is easily won back tenfold by hits. Look at the 'filter cache' and 'cached load/store queue' for similar very cheap additions with big savings.

> The problem with placing it in the pipeline or linking to instruction
> issue is that appears to be very hard to clock gate, at least a second load unit can be reasonably easily
> clock gated when not needed (or just get the store issue pipe to double duty for load as well).

Please elaborate. I don't see the above two papers mentioning any problems with clock gating.

> it would hurt FLOP/s and FLOP/watt. The P4s trace cache comes to mind.
> > > When the facts change, I might change my conclusion. Until then, I'm quite confident
> > > of where I stand. And I'm still open to that wager about the existence of the L0...
> >
> > I'm not trying to convince you that I'm correct about the L0 cache. I'm trying to convince you that
> > a dual-ported L1 is not a given due to x86 being a load-op ISA. Other x86 architectures, as well
> > as GPUs, have a lower MEM:ALU ratio and they're doing fine. But a single L1 load port wouldn't explain
> > the code Eric has posted. So I proposed the L0 cache as something which would explain all the given
> > "facts" at once. Whether that's the solution Intel ended up implementing, I don't know. It's just
> > one of the viable possibilities besides assuming it "must have" two L1 read ports.
>
> A far better explanation (if you really want to avoid two load ports) is Load elimination (especially as Intel
> actually have a patent filed for that) i.e. where identical loads are eliminated by PRF redirection during rename.
> But the only reason a complier would explicitly take advantage of such a technique is to reduce total instruction
> counts or to deal with register pressure, it would not offer power savings v normal register use.

Why would it not save power? You're saving accessing the L1 cache for the same operand.

> Given the facts (or more absence of them) and that the complier appears to issue
> similar code for both Knights landing and knights corner I think the most likely
> explanation is the complier's back end is bugged for these targets.

It amazes me that you find a bug an acceptable explanation for this horrendous code (assuming no hardware structure to efficiently support it), while the insignificant power consumption of an L0 cache is somehow unacceptable. The software is just as important as the hardware. I can assure you that no compiler developer would make even an early prototype available that would produce code like this, unless it was intentional and the result of knowledge about the architecture. It can't be due to an unimplemented optimization, because the Haswell code looks as expected and it would be trivial to use it for the AVX-512. Compiler developers look at assembly output all day long, so this isn't something that just slipped their attention.