Hi David,

David Kanter (dkanter@realworldtech.com) on 2/15/11 wrote:
---------------------------
>Nicolas Capens (nicolas.capens@gmail.com) on 2/15/11 wrote:
>---------------------------
>>David Kanter (dkanter@realworldtech.com) on 2/11/11 wrote:
>>---------------------------
>>>>That's exactly my point. The latency doesn't make or break >being thoughput-oriented.
>>>>Unless you want to imply that AMD's GPUs are less of a >throughput-oriented architecture than NVIDIA's?
>>>
>>>It's just one of many things.
>>
>>Then please sum up the "many things" that make a CPU not >capable of high *effective*
>>throughput for graphics and HPC workloads. Keep in mind >that a 6-core Sandy Bridge
>>with FMA would be capable of delivering 650 GFLOPS, while >the 12 EU IGP can only
>>do 130 GFLOPS. What sort of magic makes the latter, which >has lower theoretical
>>throughput and little or no latency optimizations, achieve >significantly better effective throughput?
>
>CPUs are designed for good throughput, but that's simply not their primary design
>target. If you don't believe me, let's look at some hard facts for single precision performance.
>
>SNB gets about 2.3GFLOP/W and 21.3GB/s memory bandwidth
>GF104 gets about 6GFLOP/mm2 and 115GB/s memory bandwidth
>Cayman gets about 10GFLOP/mm2 and 176GB/s memory bandwidth
>Cypress gets about 14GFLOP/mm2 and 160GB/s memory bandwidth
>
>That's about an order of magnitude difference between a best of breed CPU and GPU.
>You simply cannot look at that data and conclude that CPUs are equally good at
>throughput.

You're forgetting FMA and the area taken by the IGP. The CPU could have 3.2 GFLOPS/mm2. Sandy Bridge's IGP has a compute density of 3.1 GFLOPS/mm2. And that's theoretical throughput. We know AMD's architecture only achieve half of the effective throughput of NVIDIA's architecture, and even the latter can have a hard time keeping the ALUs fed. So they're not all that far apart at effective throughput.

Those discrete GPUs are also all significantly larger so no point in comparing their absolute bandwidth. Try maybe GF106 instead, it has 57.73 GB/s (and 2.5 GFLOPS/mm2). Next take into account that the CPU has large caches so it doesn't have to go to RAM that often. And last but not least, the IGP is limited to the same bandwidth as the CPU anyway.

And if we take power consumption into account we probably have to halve the frequency to approach the power consumption of the IGP, which also halves the GFLOPS/mm2. But that's fine, the total GFLOPS of the homogenous CPU is still higher. And the extra area has already been paid for anyway to make the CPU capable of running everything other than graphics as well. So we're looking at about a 2x disadvantage in effective thoughput density, but it's not stopping software rendering from becoming a viable graphics solution.

>Moreover, the reality is that the CPU in question has a pretty big
>power (and area) advantage due to using 32nm HK/MG vs. 40nm polysilicon.

Yes that's reality. By the time TSMC has mastered 28 nm Intel will be producing at 22 nm. And this gap isn't about the dissapear. But whether you like it or not, it helps the CPU to be good enough at throughput computing to take over the role of the IGP.

>Many of the design choices in CPUs will reduce perf/area, for example:
>
>1. 8T SRAMs for L1 cache
>2. High frequency operation
>3. Forwarding and bypass networks
>4. OOOE
>5. Microcode
>6. Branch prediction
>7. uop caches
>8. Coherency
>9. Large TLBs
>10. Larger SRAM cells for higher frequency
>11. Multi-ported (rather than banked) RFs
>12. Load-to-use latency from L1
>
>The list goes on...

Some of these improve effective throughput so they're not a loss. Others are vital to make the CPU capable of running the O.S. and any application, and that's largely paid for already. In comparison the GPU is utterly worthless on its own. So you really can't look at throughput density alone to assess whether or not a homogeneous architecture is viable.

>>Also note once more that a GeForce GT 420 has a TDP of 50 Watt, at practically
>>the same clock frequency as Sandy Bridge's IGP, while also only achieving 130 GFLOPS.
>>Mobile Sandy Bridge CPUs have a TDP of 45 Watt. Note that it can be further reduced
>>by clocking them lower, while still offering more GFLOPS >than the IGP.
>
>There are 4 cores in SNB, and 1 GPU. What's the area/power efficiency? The GPU
>doesn't use all 45W, even when running full throttle.

This site suggests otherwise: http://www.legionhardware.com/articles_pages/intel_core_i5_2500k_and_core_i7_2600k_sandy_bridge,14.html

>>Obviously FMA increases the power consumption of the CPU >as well, but the control
>>logic remains practically the same.
>
>Not really. It's a 3 or 4 operand instruction and requires substantially more
>work. But the area increase is less than the FLOP/s increase.

AVX already features 3 operand instructions, today.

>>So to power consumed by the control logic goes
>>down with every process node, while more of the power >budget is spent on actual
>>floating-point operations. At the same time GPUs have to >do the reverse. There's
>>no FMA to add and simply doubling the number of shader >cores does not double the
>>*effective* throughput. Instead they need to rely on >smarter scheduling or larger
>>register files and caches to compensate for the latency.
>
>Well, if you want to talk about effective throughput, then we need to come up with
>a mutually agreed upon workload that measures effective throughput. Theoretical
>FLOP/s are nice and easy to work with, albeit imperfect.

Crysis is a good starting point. Certainly way better than comparing theoretical throughput.

We should actually take future workloads into account as well. A CPU with FMA and gather/scatter can be used for a lot of throughput-oriented workloads the IGP would totally suck at. This adds value too. Even in combination with a powerful discrete graphics card, such a CPU would be much more interesting than a CPU without gather/scatter and an IGP which remains idle.

Anyhow, let's start with Crysis.

>>I believe you're greatly overestimating the power >consumption of the CPUs control
>>logic, compared to the GPUs control logic. It takes more >area to achieve ILP, that's
>>certainly true, but when you're only executing NOPs >there's practically no switching
>>activity. The power consumption is proportional to the >number of uops being processed.
>
>No it's not. Power consumption = C*F*V^2. C has to do with transistors switching
>which is only loosely correlated to uops. Some uops switch more than others.

You appear to have missed that I was talking about the control logic. The average switching activity for the control logic of different (arithmetic) uops is practically the same.

So what I was trying to say is that you pay the price of higher power consumption, when effectively achieving higher throughput. Or put another way: there are times when a GPU could be achieving higher effective performance/Watt if it spent some power extracting dynamic ILP.

It only partially compensates the CPUs power consumption, but sufficiently to make a future CPU with FMA and gather/scatter capable of taking on graphics. Note once more that the area investment would be an issue for GPUs, but with a CPU it's already paid for.

>>Dependent instruction latency is irrelevant to being >throughput oriented or not.
>
>You are 100% wrong here. The dependent integer operation latency is roughly 8X
>worse on a GPU than a CPU. That has a big cost in terms of area and power for CPUs,
>and it enables them to actually run general purpose workloads.

I'm not 100% wrong. If I was, then the GPU with the highest possible dependent instruction latency was the most efficient at throughput oriented workloads. But we both know this leads to register starvation and thus low effective throughput. Clearly you can't conclude anything from latency alone.

So how much of this 8x latency difference you claim is actually due to ALU lantecy? Note that on a CPU even an integer addition and multiplication have different latency (1 and 4), due to forwarding. So you can't simply look at the smallest latency and say the CPU is massively latency optimized, to the point where it sacrifices lots of effective throughput. If an add took 4 cycles instead, would that make the CPU 4 times more throughput-oriented? Obviously not, so likewise GPUs that spend most of the latency outside the ALUs aren't that much more throughput optimized.

Or in other words, a forwarding network massively reduces latency but that doesn't mean it required proportional area or power. And some of it is compensated by enabling the use of a smaller register file, reducing the number of threads to schedule, reducing cache pollution, increasing Amdahl's speedup factor, etc.

>>>>>Anyway, long latency instructions are the best case for >>GPUs, the reason CPUs are
>>>>>clocked at 3GHz+ and hav deep OoO buffers is the >>integer operations where GPUs are, what? 30 times slower?
>>>>
>>>>Still not proving this is relevant to being throughput-oriented or not.
>>>
>>>Actually it does. CPUs are simply the best for latency sensitive workloads. GPUs
>>>are the best for some throughput sensitive workloads.
>>
>>It's clear by now that this "30 times slower" was yet >another prejudice. So calling
>>things "the best" based on this misinformation is >worthless.
>
>Actually, it's not too far off. As you said, the latency for an Nvidia integer
>operation is about 24 cycles. So the latency of a pair of dependent ALU ops is
>roughly 48 cycles @ 1.5GHz (32nm). On a CPU, that would be 2 cycles @ 3.4GHz (~0.6ns).
>That's actually higher than 30X.
>
>Even ATI is ~20ns for dependent ALU ops.
>
>Anyway, you just need to look at the data. The conclusions are obvious.

Only incomplete or irrelevant conclusions can be drawn from looking at one data point alone. 30x higher dependent instruction issue lantency is not a feat by any means and can't be used to determine effective throughput performance.

The viability of software rendering for a CPU with FMA and gather/scatter is affected by many more aspects as a whole. Analysing the data in depth shows that the 'gap' is nowhere near an order of magnitude as you're trying to imply, and the value of a homogenous architecture is greater than that of a CPU+IGP. Plus things are converging as well.

Take care,

Nicolas