Hi David,

David Kanter (dkanter@realworldtech.com) on 2/11/11 wrote:
---------------------------
>>That's exactly my point. The latency doesn't make or break >being thoughput-oriented.
>>Unless you want to imply that AMD's GPUs are less of a >throughput-oriented architecture than NVIDIA's?
>
>It's just one of many things.

Then please sum up the "many things" that make a CPU not capable of high *effective* throughput for graphics and HPC workloads. Keep in mind that a 6-core Sandy Bridge with FMA would be capable of delivering 650 GFLOPS, while the 12 EU IGP can only do 130 GFLOPS. What sort of magic makes the latter, which has lower theoretical throughput and little or no latency optimizations, achieve significantly better effective throughput?

Also note once more that a GeForce GT 420 has a TDP of 50 Watt, at practically the same clock frequency as Sandy Bridge's IGP, while also only achieving 130 GFLOPS. Mobile Sandy Bridge CPUs have a TDP of 45 Watt. Note that it can be further reduced by clocking them lower, while still offering more GFLOPS than the IGP.

Obviously FMA increases the power consumption of the CPU as well, but the control logic remains practically the same. So to power consumed by the control logic goes down with every process node, while more of the power budget is spent on actual floating-point operations. At the same time GPUs have to do the reverse. There's no FMA to add and simply doubling the number of shader cores does not double the *effective* throughput. Instead they need to rely on smarter scheduling or larger register files and caches to compensate for the latency.

There's no escaping the convergence. You can't have a million shader cores without a massive amount of control logic and on-die storage to keep feeding them independent data. Just look at the history of GPUs. The first generation had practically no control logic and no on-die storage. Data went in at one end of the pipeline, and the processed data coming out was written back to RAM. Nowadays we have a massive number of strands in flight. This requires lots of control logic and storage to hold the context for each of them.

Note also that unification helps combat this. By turning texture units into additional generic gather/scatter units and performing the filtering in the shader cores, the (expanding) shader array has more work during heavy filtering, is less bottlenecked due a lack of generic load/store operations, and offers more performance during arithmetic limited tasks. You get these benefits in exchange for a little bit of extra power consumption, but that gets amortized by smaller process nodes anyway.

I believe you're greatly overestimating the power consumption of the CPUs control logic, compared to the GPUs control logic. It takes more area to achieve ILP, that's certainly true, but when you're only executing NOPs there's practically no switching activity. The power consumption is proportional to the number of uops being processed. And I've also already detailed how gather/scatter support would reduce the number of uops for parallel load/store operations by a factor 8-24. As for the area cost, that's fine, because the consumer is already paying for that anyway to make the CPU efficient at latency-oriented workloads.

So seriously, the question isn't whether or not CPUs will take over the role of the IGP, but when.

>And honestly, latency between NV and ATI is pretty
>similar. If you look at it quantitatively - comparing the latency of dependent
>operations, you'll see they are vastly more similar to each other, than a CPU.

Really? Cayman XT takes 8 cycles at 880 MHz. So that's 9 ns. GF110 is reported to have a warp latency of 32 cycles, and clocks at 1544 MHz. So that's 20 ns. Clearly a factor 2x is not enough to call one or the other sigificantly worse at the relevant throughput-oriented workload: graphics.

That said, Cayman takes 4 cycles to read the register file. So the ALUs themselves take 4.5 ns. And the ALUs can perform dot products and even SFU operations. A dot product instruction takes 12 cycles on the CPU, so at 3.4 GHz that's 3.5 ns. Clearly CPUs don't have to push their ALUs very hard.

Dependent instruction latency is irrelevant to being throughput oriented or not. In fact you'd better have a low lantecy to prevent running out of registers. Note that executing dependent dot products on GF110 even takes 62 ns. Clearly it's not making or breaking a graphics architecture though.

At the thing that actually could be directly relevant to efficiency, ALU latency, CPUs are not significantly different. So I'm sorry but I think you got fooled by your own preconceptions again.

>>>Anyway, long latency instructions are the best case for GPUs, the reason CPUs are
>>>clocked at 3GHz+ and hav deep OoO buffers is the integer operations where GPUs are, what? 30 times slower?
>>
>>Still not proving this is relevant to being throughput-oriented or not.
>
>Actually it does. CPUs are simply the best for latency sensitive workloads. GPUs
>are the best for some throughput sensitive workloads.

It's clear by now that this "30 times slower" was yet another prejudice. So calling things "the best" based on this misinformation is worthless.

That said, yes, CPUs are great for latency sensitive workloads. But that's not preventing them from being good at throughput sensitive workloads! Getting the best of both worlds takes area, but as noted before the register file reduction compensates that somewhat, and in a CPU it's paid for anyway. So unifying the IGP into the CPU makes a great deal of sense.

>>>>Note though that on GT200 it was 24 cycles, so there's some convergence taking place.
>>>
>>>In GT200 it was pathetically slow, not being so slow doesn't prove anything,
>R600 still have lower latency than Fermi.
>>
>>Again proving latency optimizations don't make it any less >of a throughput-oriented
>>architecture.
>
>Yes it does. Unlike software, hardware is a 0-sum game where spending power or
>area for an optimization means it isn't available for another. Sometimes you can
>save power with an optimization, but you spend area. Sometimes you save area, but
>spend power. Regardless, you have finite engineering resources, which means you
>have to choose whether to spend more on latency optimization or throughput optimization.
>
>If you are spending a lot of power/area to reduce latency, you are not by definition
>NOT spending that power/area to improve throughput. Reducing latency can improve
>throughput, but it's not the optimal strategy. It improves flexibility, but again,
>is not optimal for streaming type workloads.

Wrong again. Spending area to reduce lantecy does not have to mean "by definition" not improving *effective* throughput. Reducing latency means you don't need to keep as many strands in flight, which in turn means you can have a smaller register file and the cache pollution is greatly reduced.

As an historic example, note that decoupling the GPU's execution of arithmetic instructions and texture instructions, was effectively a latency reduction optimization. It costed control logic but this latency optimization was the key to allow an increase in throughput without costing a gigantic register file.

So with all due respect, calling hardware design a zero-sum game is plain incorrect. There are design choices which optimize multiple aspects, increasing effective performance beyond what's possible by focussing on optimizing one aspect alone.

Even if we consider software to be less of a zero-sum game than hardware, it's clear that this is an argument pro unification.

Best regards,

Nicolas