Hi Eduardo,

EduardoS (no@spam.com) on 2/10/11 wrote:
---------------------------
>Nicolas Capens (nicolas.capens@gmail.com) on 2/9/11 wrote:
>---------------------------
>>Indeed. You appear to have missed the word "aggressive". At 90 nm, 3.8 GHz was
>>really pushing it. Actually, the Pentium 4 had an integer core at twice the frequency.
>>It required very short pipeline stages and Low-Voltage Swing circuits, which takes
>>a lot of extra transistors. So at 32 nm, 3.8 GHz is a breeze.
>
>Clock frequency != latency
>
>>The latency of a multiply-add on GF114 is 18 cycles I believe. On Sandy Bridge
>>it would take 8 cycles. That's roughly a 2x difference, or 4x in absolute time.
>
>In AMD GPUs it's 8 clocks, in absolute terms it's lower than GF114, clock frequency
>!= latency, remember that? and still 3 times higher than Sandy Bridge, but the latter
>don't does MADs, Bulldozer will do, in 6 clocks.

That's exactly my point. The latency doesn't make or break being thoughput-oriented. Unless you want to imply that AMD's GPUs are less of a throughput-oriented architecture than NVIDIA's?

>Anyway, long latency instructions are the best case for GPUs, the reason CPUs are
>clocked at 3GHz+ and hav deep OoO buffers is the integer operations where GPUs are, what? 30 times slower?

Still not proving this is relevant to being throughput-oriented or not.

>>Note though that on GT200 it was 24 cycles, so there's some convergence taking place.
>
>In GT200 it was pathetically slow, not being so slow doesn't prove anything, R600 still have lower latency than Fermi.

Again proving latency optimizations don't make it any less of a throughput-oriented architecture. And when looking at the GPU generations from one vendor, the latencies are going down while for CPUs the latency remained nearly constant, proving there's convergence taking place. Comparing the latencies for different GPUs (with significantly different clock frequencies) isn't valid at all so it doesn't disprove this. The real reason they differ is because a higher clock frequency requires more pipeline stages thus more latching latency. And the reason CPUs have lower latency despite higher clock frequency is because forwarding makes the execution pipeline shorter.

With that taken into account the latency of CPUs is still somewhat lower, but not very sigificantly so, and as pointed out there's still convergence taking place.

>>Furthermore, the latency of the CPU is based on making use of argument forwarding,
>to bypass the register file. On a GPU the latency includes accessing the register
>file, which for AMD's architecture takes half of the total latency (base on what you wrote about AMD Cayman here:
>
>I didn't read the article neither care about it (sorry DK), the important thing
>is, Cayman DOES forwarding, it's explicity but it does, it does not access the register
>file (I mean, the big SRAM banks) for dependent instructions.

According to David's article, there are "virtual" registers (let's call the other ones stored in SRAM "context" registers), but there's no mention these virtual registers halve the lantecy for dependent instructions (or equivalently; increase the available context registers per wavefront).

Possibly the forwarding you talk about is the one explained on the next page of David's article (http://www.realworldtech.com/page.cfm?ArticleID=RWT121410213827&p=6). This is forwarding between pipelines, not back to the top of the pipeline for the next instrution, but for the same instruction. It allows "horizontal" operations like dot products, which takes higher latency than multiply-add. Sandy Bridge supports dot products too, and it takes 12 cycles (for comparison, add takes 3 and mul takes 5).

Once more this proves that at the ALU level the latency difference is nowhere near as big as David and you would like to believe. CPUs are more latency optimized, but they're not aggressively latency optimized, in the sense that it sacrifices a significant amount of effective throughput. As a matter of fact, the CPUs lantecy optimizations allow it to execute workloads which would cause the GPU to run out of registers, much more efficiently. So the CPU is a much better throughput-oriented processor for these workloads. Unfortunately for aggressively throughput-oriented GPUs, graphics workloads continue to get more complex and GPGPU is even worse so they can't ignore latency optimization.

Convergence is the word.

>>In conclusion, CPUs are indeed latency optimized, but that doesn't mean it can't
>be a throughput oriented architecture.
>
>Id means too much transistors are spent to reduce latency, with many transistors
>on few units one can't push many units.

Why would this be "too" many transistors? Reducing latency improves effective throughput for complex workloads. Effective throughput is the only thing that matters. Focussing too aggressively on increasing the ALU count leads to being latency bound, meaning "too" many transistors are spent on it.

Regardless of whether it's arithmetic latency or memory latency, GPUs continue to have to invest transistors into latency reduction. At the same time, CPUs slow cache/core growth, widen the vectors, add FMA, add SMT, etc. The word.

>>You missed the point. There's a design space for compute oriented devices, which
>>includes a wide range of clock frequencies (3.2 GHz for Cell BE at 90 nm). So you
>>can't conclude that modern x86 CPUs aren't compute oriented, based on their clock frequency (or latency).
>
>Cell BE? That ugly failure? It was developed in another context, it means nothing today.

This "ugly failure" hasn't stopped the PlayStation 3 from selling on average a million units a month. And despite the aging the sales numbers are still on an upward slope. So much for not meaning anything today.

You could argue that there's more to the PlayStation 3 than Cell BE, but it doesn't prove Cell isn't a throughput-oriented architecture. You'd expect sales to suffer if the CPU was preventing it from being a succesful gaming and multimedia platform.

And in case you consider it an "ugly failure" because of the heterogenous programming model, that too proves one of my points. Developers prefer homogenous architectures.

>>Unless of course you want to imply that NVIDIA's GPUs are any less of a GPU due to their higher clock frequency?
>
>Clock frequency != latency, got it?

Yes, I got that all along. Chickens are not eggs either, but that doesn't mean they're unrelated.

Besides, it's David who first mentioned both frequency and latency as arguments for being throughput oriented or not. The fact of the matter is that today's GPUs have different frequencies and different latencies but they're all considered throughput-oriented. There's a pretty wide design space where you can trade latency for theoretical throughput (potentially involving frequency changes), without affective *effective* thoughput for a mix of workloads.

CPUs are slightly outside of that design space for today's graphics workloads, but there's still FMA (doubling arithmetic throughput) and gather/scatter (massively improving load/store throughput), and the workloads are getting more complex, shifting the design space in favor of the CPU.

There are strong forces at play here wich make software rendering viable in the foreseeable future. The word.

>>I never denied CPUs are latency optimized! But they're no longer "aggressive" about
>>it (i.e. they don't sacrifice everything else for the sake of latency reduction).
>
>But they sacrify a lot!

Like what? Without any argumentation this is purely a preconception.

Yes they scrifice theoretical throughput. But that's utterly irrelevant compared to effective throughput. As I've shown before GPUs also already sacrifice considerable theoretical throughput (massive register files and caches) in order not to see the effective throughput plummet. NVIDIA even sacrifices half the FLOPS compared to AMD and still produces a highly competitive (if not superior) graphics chip. So saying CPUs sacrifice a lot is meaningless without detailed analysis.

And once more you have to take convergence into consideration for such an analysis. Gather/scatter support for CPUs is still several years out, so how much does such a future CPU sacrifice, compared to future GPUs? That's the question you should answer to prove CPUs are not throughput-oriented enough to handle graphics.

>>The MHz-race is over.
>
>Clock frequency != latency, pushing clock frequency too high can actually increase
>your latency and CPUs are designed to reduce the latency!

Certainly, but you're not proving that the clock frequency of future CPUs is "too" high to achieve a good effective throughput.

That's why I said the MHz-race is over. GPUs clock frequencies keep increasing at a faster pace than CPUs. The word.

>>It has made way for a focus on effective performance/Watt,
>>leading to a healthy balance between latency and throughput optimization.
>
>Want's performance/Watt? Shrink a 8086 (hum... 6502?) to 32nm, but few would buy
>it, energy still cheap and business have goals that an 8086 can't achieve.

Next to cost, effective performance/Watt is everything. Pick a price (read: die size), a TDP, and performance/Watt is what sets architectures apart.

You can also do the reverse: set a performance goal and see how cheap you can make it to conquer a bigger market share than the competition. You can use a smaller die by clocking it higher, till you get close to the maximum TDP acceptable for the system you're creating. Again performance/Watt is the key factor.

So with all due respect, your example of shrinking an 8086 is pretty silly (and I mean silly aside from the fact that it's meant to be a silly example). If you take cost and acceptable TDP into account, you need another architecture. One with the best performance/Watt.

>>Hold your horses. The scheduling efficiency of the VLIW5 core was 3.4 operations
>>(http://www.anandtech.com/show/4061/amds-radeon-hd-6970-radeon-hd-6950/4).
>
>Blame the compiler, unless the code relies too much on the T unit or is too branchy
>(in this case it would sucks on GPUs anyway) there is no reason for lower than 80%
>packing on such data parallel workloads.

What's your point, aside from proving mine?

VLIW isn't the limiting factor. NVIDIA's architecture also has low ALU utilization when you stuff the code with SFU operations or branches. But while NVIDIA's utilization isn't 100%, AMD's overall utilization is only half of it!

So clearly AMD's architecture is bottlenecked by other things, very often. The only reason they have this many ALUs, is because they compensate for the bottlenecks by burst executing workloads that aren't bottlenecked. There's nothing really wrong about that for today's graphics; noone says you need to be compute limited all the time to achieve high effective graphics performance. But they're in trouble for more complex workloads, and going forward this architecture doesn't scale well (when you keep the ratios the same). Even for graphics the parallism isn't infinite, so they need to increase the investment in things other than theoretical throughput.

>> So the
>>move to VLIW4 made it very efficient. Also note that instead of one fat ALU taking
>>the role of SFU, three ALUs can together operate as an SFU. This unification means
>>there's even better scheduling opportunity.
>
>The SFU is fat, stupid fat and takes a lot of space, it takes three to five multipliers
>inside of it, when doing an MUL or ADD most of the fat (I mean, stupid fat) will
>be under-utilized, just taking space from other units that could be doing more MULs
>or ADDs, take away the fat unit and put two or three slim units in it's place, it's
>a throughput oriented move, reducing compiler inefficiencies is more a side effect than a reason.

Taking the SFU away is clearly not an option. So you need to ensure that the fat doesn't weigh down the architecture too much. The solution they found was to unify it with three slim ALUs, which has the added effect of better VLIW scheduling. If the latter wasn't a reason and just a coincidental benefit, then why didn't they keep the fifth ALU?

It can't be SFU:ALU instruction ratio. Fermi has a fixed 1:4 and 1:6 ratio depending on the model, and uses the SFU for interpolation as well. AMD's SFU:ALU ratio can range from 1:1 to 0:4. It could have been 1:2 - 0:5 with an extra ALU if scheduling efficiency wasn't an intended reason. The average code would need to have less than a 1:2 ratio to prefer the former over the latter, but then NVIDIA would use a 1:2 ratio as well, especially since it uses the SFU for interpolation as well.

Even if you factor in T-unit and load/store unit usage, NVIDIA would be better off with less ALUs if a fifth slim ALU didn't make sense for AMD.

Clearly something else is at play which makes the fifth ALU not worthwhile. So it must have been intentionally removed to improve VLIW scheduling efficiency, or you're implying the AMD engineers are stupid.

>Oh... Unfortunally for AMD their engenieers weren't able to significantly improve
>the number of ALUs, I wish them better look on the next try.

Ok, you're implying they're stupid then.

Given that they *started* from a VLIW5 architecture, it seems more likely to me they evaluated keeping the fifth ALU and just unifying the SFU, but decided to reduce the width as well to improve scheduling for workloads containing ever less explicit parallelism.

Note that despite a high demand for floating-point performance in HPC computing, both Intel and AMD never had any CPU with more than two floating-point execution ports (not even Itanium). This should tell you something about the scheduling difficulties of VLIW5, and the direction AMD is trying to take (complex HPC workloads). The word.

>>So it would be pretty ridiculous if low scalar memory access performance alone
>>was responsible for lowering the efficiency to 50% of that of NVIDIA's architecture.
>>You'd think someone would have noticed that and fixed it by now.
>
>I don't know what happen at AMD but by some projects it's clear some there are
>aiming at better balance, some fixes take time...

There is no need to fix anything. They could have simply kept the fifth ALU. But they consciously tossed it out.

>BTW, nVidia will push their raw power on next generation, while not all this part
>is very important for throughput optimized architectures.

Obviously when moving to 28 nm they'll increase ALU count. But it will take a major amount of the extra transistor budget, to fight Amdahl's Law and keep these ALUs fed with data.

Cheers,

Nicolas