Hi David,

David Kanter (dkanter@realworldtech.com) on 2/25/11 wrote:
---------------------------
>Nicolas Capens (nicolas.capens@gmail.com) on 2/24/11 wrote:
>---------------------------
>>>>To hide the latency of long dependency chains you need software pipelining, but
>>>>this increases register pressure (software pipelining uses architectural registers,
>>>>so renaming doesn't help here). FMA reduces the latency of the critical path a bit
>>>>so less software pipelining is needed, keeping register pressure low.
>>>>
>>>>Granted, the effect is small, but FMA is a bigger win on >register-impaired architectures like x86.
>>>
>>>Except that you don't have software pipelining on x86.
>>
>>Software pipelining does not require architectural >support. It's a reordering of
>>the instructions of a loop to take advantage of >independent iterations. This
>can be done manually or by >the compiler.
>
>I'm sorry, I made a terminology mistake. I wrote "SW pipelining", but really meant
>"HW features for SW pipelining". I was thinking of the various stuff in IA64 missing from x86.
>
>You can do SW pipelining on x86, but you may actually lose performance due to the
>uop cache in SNB (and loop caches in previous designs). Moreover, SW pipelining
>is less valuable on x86 because OOOE will take up the slack, and the ISA lacks the
>supporting features that IPF has (rotating registers, predication, etc.). That
>reduces the effectiveness of your SW pipelining and bloats up the code foot print.
>
>You are correct that SW pipelining does work for x86, but it's just not as big
>a win. Especially if you blow out the uop cache (when it could be a big loser).

In my experience software pipelining can be a significant win even for x86, because it breaks up dependency chains. And it looks like I'm not alone: http://www.springerlink.com/content/3485p7q146261803/

Sandy Bridge's uop cache is 1536 entries big, which the vast majority of software pipelined loops won't exceed. And even when it is exceeded, it's not that big of a loss. You've worded it quite nicely in your own Sandy Bridge review: "The uop cache was carefully designed so that it is an enhancement for Sandy Bridge and does not penalize any workloads."

>>>Instead, you have a large
>>>pool of rename registers - even larger than IPF's >>register file. So it's actually a non-issue.
>>
>>Register renaming merely helps avoid false dependencies >with out-of-order execution.
>>It doesn't help software pipelining.
>
>Sure it does. Multiple iterations of a loop can fit inside the OOO window, at
>which point they can be freely re-ordered and renamed to the same effect. Of course,
>if it's a big loop...it may not fit in the OOO window.

Exactly. The loops that benefit from software pipelining have long dependency chains that don't fit in the ROB. Register renaming doesn't help here.

>>It generally requires the use of more registers
>>in the actual code, and thus can cause unwanted spilling. >FMA may avoid some of that.
>
>This is entirely a tangent. FMA does not dynamically reduce the number of register
>reads and writes (which is what matters for power). In an OOO design, the result
>of the multiply is forwarded to the add, so they have the same number of register
>reads and writes. Moreover, by virtue of the renaming the extra ISA register needed
>for separate multiply and add is really a non-issue.
>
>The benefit is primarily in getting more compute per instruction.

Yes, I can live with that.

>>>>>If anything, FMA could only indirectly increase register pressure, because FMA-capable
>>>>>FPU is likely to have higher accumulator-to-result latency than dedicated FP_ADD unit.
>>>>
>>>>It appears to me only two gate delays are required to pass >the operand directly
>>>>to the adder.
>>>
>>>In most FMA architectures, an add is implemented by >>multiplying by 1. As it turns
>>>out, multiplying by 1 still takes cycles.
>>
>>You can instead just forward the operand directly to the >adder part (and gate the multiplier clock to save power).
>
>That's correct.
>
>>>>Since the current floating-point addition instruction >>>has >a 3-cycle
>>>>latency, this doesn't seem significant to me.
>>>
>>>What FMA instruction has a 3 cycle latency? I don't know >>of any architecture with a 3 cycle FMA latency...
>>
>>You don't need a 3 cycle FMA lantecy to get a 3 cycle ADD latency: http://www.acsel-lab.com/arithmetic/arith17/papers/ARITH17_Bruguera.pdf
>>
>>And here's an interesting article by AMD which hints at the FMA implementation they may use for Bulldozer: http://www.docstoc.com/docs/22731574/Bridged-Floating-Point-Fused-Multiply-Add-Design
>>
>>>
>
>I'm familiar with the work. The real question is how it turns out in silicon.
>You seem to believe that a 3 cycle FADD latency is feasible on such a design.
>I don't. I'm quite confident that Bulldozer will have more than 3 cycles of latency for FMA, FADD or FMUL.
>
>Increasing the latency is not great for performance and just goes against the prediction you made earlier.

Some sources indicate that the latency has been increased to 6 cycles for all these operations. But while this is indeed not ideal it can be hidden quite effectively with software pipelining. Also note that 6 cycles for FMA is still lower than 4+4 cycles on previous architectures. Furthermore, the research suggests there's still room for improvement.

>>>>FMA support will be added on the 22
>>>>nm process (some say it may arrive later) while the clock >frequency won't increase
>>>>by much due to power consumption, meaning there's more >headroom.
>>>
>>>Why would you think that? Intel and AMD engineers aren't >>idiots. If they had
>>>headroom, they would trade it for lower power.
>>
>>Indeed, they would trade it for lower power, except for >the critical paths.
>
>Critical paths in one part of the chip do not prevent engineers from using timing
>slack in other paths to decrease power.
>
>Intel, Oracle, Nvidia, IBM and AMD (pretty much everyone in fact) aggressively
>eliminate slack in timing paths (where possible) to lower dynamic power consumption and leakage.
>
>I'd like to point out that you came into this discussion claiming that you had
>a good understanding of hardware design. Yet you are making statements like this
>that are obviously false - it doesn't do much for credibility.

I'm saying the exact same thing as you!

>>The
>>end result is still a much more power efficient design but >with higher throughput and equal latency.
>
>Except what most of us are saying is that the latency will be higher.

That's just one implementation. And AMD may have traded latency for even better power efficiency. It's hard to tell what tradeoffs they made without having the actual hardware. In any case FMA support should overall be a significant win in performance/Watt.

>>It's not all or nothing. There are many interesting >compromises that can be made. In any case the CPU is >converging closer to the GPU in terms of compute density. >And other studies conclude that GPUs need result forwarding to handle more workloads efficiently: http://www.ece.wisc.edu/~wddd/2009/papers/wddd_01.pdf
>>Double convergence at the hardware level, and convergence >at the software level too.
>
>Just because it's written in an academic paper doesn't mean it will happen in real silicon.

That's not a counter-argument, just a generalization.

>>>>Even if it prevents
>>>>them from reducing the latency to 2 cycles, FMA won't make >things worse compared to today's situation.
>>>
>>>2 cycles for what?
>>
>>2 cycles for an isolated ADD unit due to the process headroom, which could be impossible
>>for an FMA unit with operand forwarding due to the extra gate delays. Anyway, it's
>>highly unlikely they had plans to aggressively reduce the latency anyhow, and would
>>save power instead. Even with the extra gate delays I >expect it to still save power,
>>double the throughput with FMA, and keep the ADD latency >the same.
>
>You've made a prediction, that Bulldozer will have a 3 cycle latency for the FMA/FADD/FMUL.
>This prediction is based upon your understanding of HW design.

I did not make a prediction. I laid out the theoretical possibilities. There can be many reasons why the effective latency ends up higher. They may for instance aim for high clock frequencies while keeping the power consumption acceptable. Or they may simply not have had the budget for a more optimal first implementation. That doesn't mean the theory for FMA is false.

Cheers,

Nicolas