Nicolas Capens (nicolas.capens@gmail.com) on 2/8/11 wrote:
---------------------------
>Hi David,
>
>David Kanter (dkanter@realworldtech.com) on 2/7/11 wrote:
>---------------------------
>>Nicolas Capens (nicolas.capens@gmail.com) on 2/7/11 wrote:
>>---------------------------
>>>Hi David,
>>>
>>>David Kanter (dkanter@realworldtech.com) on 2/7/11 wrote:
>>>---------------------------
>>>>The fundamental point is that efficiency *really* does matter a lot. In a world
>>>>where power is the #1 performance limiter, doing stuff in dedicated hardware becomes
>>>>increasingly attractive. If you can cut the energy required to render a frame in
>>>>half by using an IGP (which is quite believable), then you have gained a huge amount
>>>>of energy to expend in the CPU or other areas.
>>>>
>>>>The key is understanding that with Moore's Law, spending more transistors to lower
>>>>power is VERY attractive. That dictates more dedicated hardware, not less. Just
>>>>look at what Intel has done - putting embedded micro-controllers inside CPUs to manage power.
>>>>
>>>>The whole idea of eliminating throughput oriented cores goes against that entirely.
>>>
>>>Who's talking about eliminating throughput oriented cores? I'm talking about adding
>>>gather/scatter to CPU cores so they *become* efficient throughput oriented cores
>>>and the functionality of the IGP can be unified.
>>
>>You are talking about eliminating throughput cores. You claim to have a basic
>>understanding of hardware, so it should be readily apparent how CPU cores and throughput
>>cores (e.g. Niagara, GPU shaders) differ.
>
>Throughput-oriented is a term which originates from server systems, long before
>graphics chips were even called GPUs! It merely means a focus on data rate, potentially
>at the cost of latency. Any multiprocessor system, is a throughput oriented system.
>Clock frequency and ILP are latency-oriented, while DLP and TLP are throughput-oriented.
>
>Today's x86 CPUs exploit DLP and TLP (SIMD, multi-core and Hyper-Threading) and
>have a less aggressive clocking than several years ago.

Actually the clock speeds are about the same 3-4GHz.

>So they are definitely throughput-oriented
>architectures already and would become more efficient at with the addition of gather/scatter
>support. Parallel load/store is the main thing setting them apart from GPUs.

That's simply false. There are many more differences in terms of circuit design and the latency of individual instructions.

>GPUs are, in the words of NVIDIA's chief scientists, "aggressively throughput-oriented
>processors". Note though that GF104 features superscalar >execution, intended to
>lower the latency. And aside from reducing bandwidth, caches also reduce latency.
>So GPUs are forces to become less aggressive at using throughput-oriented techniques,
>because reducing latency somewhat also reduces the amount of on-chip storage you
>need. It's a balancing act, because obviously reducing >latency costs transistors as well.

The two architectures are leagues apart in terms of latency? What is the latency of a dependent chain of adds in a GPU? What about a CPU? What is the branch coherence? What is the memory latency?

The memory latency makes this especially obvious, as there is a huge difference.

>>Even to someone without circuit design
>>expertise, it should be blinding obvious - the clock speeds are about a factor of 2-4X different.
>
>Then why did NVIDIA decide to put its aggressively throughput-oriented cores into
>a higher clock domain? The GeForce GTX 560 Ti has a shader clock of 1645 MHz, while
>the Radeon HD 6950 has a clock of 800 MHz. Does that mean >NVIDIA's architecture is not throughput-oriented?

1.5GHz is still half the speed or less of a modern CPU. AMD has GPUs that run at 900MHz. And those speeds have stayed constant for the last several generations (since about 65nm).

>Clock speed alone isn't an indication of being throughput-oriented or not. It's
>a design decision which doesn't have to compromise *effective* throughput, as proven
>by NVIDIA vs. AMD. Another example is Cell BE, which clocks at 3.2 GHz but even
>at 90 nm was considered strongly thoughput-oriented. x86 CPUs have come a long way
>since the single-core Pentium 4 days, and they're not about to stop increasing throughput
>efficiency (AVX, Bulldozer, FMA, etc.).

Listen, no matter what you say, CPUs are still optimized for latency. It's blatantly obvious if you have ever written code for a CPU. It's also blatantly obvious that GPUs are not optimized for latency. This means different circuits, different architectures, pipeline arrangements, etc. etc. and most importantly a different style of programming.

Consider the performance degradation of running a latency sensitive workload (e.g. SPECint) on a GPU vs. a CPU. It's going to be huge.

>As a matter of fact the CPU's clock frequency has remained nearly constant. The
>i7-2600K has a 3.8 GHz Turbo Mode ( but under multi-threaded workloads it's not
>that high), the same as the 2004 Pentium 4 Prescott. In the same time period, NVIDIA's
>GPUs have increased their clock frequency by over a factor 3. There's no indication
>this is going to change. For NVIDIA to conquer the HPC market, it needs to continue
>investing into latency reduction. To prevent an excessive growth in die size, it
>needs to increase the clock frequency. GF100 had some thermal issues, but they got
>that under control with GF110, which has more cores >enabled and even higher clocks.

Again, frequency is just one aspect. There are so many other things to consider. Pipeline depth, result forwarding, dependent instruction latency, etc. etc.

>So while it's "blinding obvious" that there's a clock frequency difference today,
>it's also "blinding obvious" they're on a collision >course. Gather/scatter support
>is still several years out, so by that time they'll have >converged even closer and gather/scatter is the keystone.

Scatter/gather is helpful, but it will not make CPUs as efficient as GPUs. It doesn't help make the ALUs lower power. It doesn't help make the front-end lower power.

Look at what CPUs have that GPUs do not:
Bypass networks
Branch predictors
Prefetchers
Out-of-order execution
Low latency instructions

etc. etc.

The differences at the circuit level are just as big.

>>You cannot simply tack on scatter/gather to a latency optimized CPU core and expect
>>it to look like a throughput core in terms of power efficiency. At least, there
>>is definitely a lack of evidence for any such claims. Moreover, you need to preserve
>>the power efficiency for workloads that cannot be vectorized.
>
>An architecture which balances latency and theoretical throughput, can still achieve
>high effective thoughput. It's how NVIDIA achieved to >outperform AMD with only half the FLOP density.

That's so wrong it isn't even funny. That's because of AMD's use of static scheduling for their VLIW, and because Nvidia is much more optimized for scalar memory accesses. Has nothing to do with latency vs. throughput.

>The way things converge, tacking on gather/scatter support >does put the GPU within
>striking distance, starting with the IGP. For someone not >time, a balanced homogenous architecture is the most cost >effective solution for all his processing needs.

I don't believe that a homogeneous architecture is optimal at all, and you have yet to show that in any meaningful way. In fact, you have admitted that it is sub-optimal for power consumption...which means that as long as graphics consumes a non-trivial amount of power, that an IGP will be a superior solution. If there is a day when graphics is merely 1-2% of all cycles, then perhaps it might happen...but I don't see that ever happening.

>Note that widening the vectors amortizes the cost of >things like out-of-order execution.
>At the same time, AMD has reduced its VLIW width from 5 to >4, in order to achieve
>higher efficiency.

So what?

>GPUs also introduced concurrent kernel execution and >scalar execution,
>and have growing register files and caches. So they're >investing more transistors
>into latency reduction and programmability than raw FLOPS. >GF110 has a 0.52 FLOP/transitor
>ratio. With G92b that was still a 0.94 ratio.

Register file size is increased for better throughput...the registers per vector lane have been decreasing.

What you are saying is obvious - GPUs are becoming more programmable. But the reality is that they are not even remotely optimized for latency. Where do you compile your code? On a CPU or a GPU?

The bottom line is that to achieve optimal throughput you must sacrifice latency (look at the memory subsystem), and vica versa. You can refuse to believe this, but it's simply true. While GPUs may become more programmable, this is all relative to an architecture that started with 0 programmability. The gap between GPUs and CPUs may shrink, but it will never disappear and the efficiency differences will always be sizable.

>It's easy to see where your preconceptions come from though. NV40 had a 0.24 ratio,
>which G92b increased by a fourfold in a few years time. But you got fooled into
>thinking that this is a trend which can be sustained. Widening a component only
>increases the overall throughput desity of that component till it reaches 50% of
>the die area. And the components themselves get fatter to increase programmability
>as well, and the rest of the architecture needs to support >the same throughput.

Of course it's easy to see where my preconceptions come from. It's reality.

>So for your own sake stop staring yourself blind at >theoretical thoughput. There's
>a lot more to effective performance than that.
>
>>>If dedicated hardware was the universal answer, then GPUs >would still have separate
>>>vertex and pixel pipelines. So clearly you're not taking >all the of the factors into account!
>>
>>Newsflash: GPU shaders are dedicated hardware. The fact that the workloads for
>>vertex and pixel shading are similar enough to use the same hardware is not relevant.
>
>That's old news. Shader cores are no longer dedicated. NVIDIA now calls them CUDA
>cores instead, for good reason. The floating-point operations are IEEE-754 compliant,
>just like on a CPU.

And what is the latency of those operations? The latency of forwarding between dependent ops? What is the branch prediction latency? What is the mispredict penalty? What is the latency of a load?

>A Tesla card is as dedicated to graphics as it is to >biochemical
>research.

You notice how Tesla is dedicated for throughput oriented tasks, rather than latency ones?

>A CPU with gather/scatter would be just as dedicated to those. The only
>things making the GPU dedicated to gaphics, are the texture samplers, ROPs, and
>rasterizers (each of which are also optimized in a >software renderer when there's gather/scatter support).

Then there's the shader cores, which are optimized for throughput rather than latency (and ditto for the memory hierarchy).

>And please explain to me why the workload is not relevant to unification, instead
>of handwaving. If pixel shading still only consisted of integer operations, there
>wouldn't be any significant reason to unify them, and shader cores would be truely
>dedicated to graphics. Unification is strong proof that >dedicated hardware is not
>the universal answer. If you still think it's irrelevant >to the discussion, please elaborate.

I already explained, so perhaps repeating myself will help. They are DEDICATED TO THROUGHPUT ORIENTED WORKLOADS.

>>Let's take another example. Icera makes a very cool SDR. However, to meet the
>>performance and power efficiency requirements, they use a custom designed chip to
>>run the SDR. So, the 'dedicated hardware' is used by many different radio protocols,
>>in exactly the same way that GPU shaders are used by many different shader types.
>It's still dedicated hardware though.
>
>Does Icera's SDR support IEEE-754? I guess not, so *this* >is irrelevant.

What does IEEE-754 have to do with latency?

>It's nothing personal, but face it, you're running out of >arguments and start handwaving
>and reaching for absurd examples which I'm easily able to >debunk.

Only because you totally fail to understand and refuse to acknowledge reality.

>>In case you haven't noticed, modern CPUs are filled with >>idle silicon. Floating
>>point units, AES crypto blocks, virtualization support, >>real mode support, etc. Many of these were added recently.
>
>Floating-point is useful to graphics, so this isn't an argument against software rendering.
>
>As for AES, virtualization, real mode, etc. they certainly don't "fill" the CPU
>with idle silicon.

Microcode?

>Unless you can prove me otherwise, AES doesn't take die space
>proportional to the GPU's, ROPs, texture samplers or >rasterizers.

The ROPs are used for general purpose workloads, as are the texture sampling units. Where do you think loads and stores are executed? And atomic operations? The rasterizer is not useful for general software, but how much power does it consume? How much area?

>And like I said
>before, fast AES support is important for generic encrypted disk and network access,
>and gather/scatter speeds up software AES so the dedicated >hardware can be removed.

You said that, but you're wrong. You cannot remove it for compatibility reasons, and also for security reasons.

>VT-x and real mode are even supported by Atom cores, so >it's doubtful this takes
>any noticable die space on a desktop chip, and it's >obviously indispensable for the software that make use of >it.

Why is virtualization support in hardware? VMware was doing fine with their binary translation. Maybe it was added to improve performance and efficiency!!!! Just like rasterizers!

>Besides, like I said before GPUs also have lots of programmability features which
>may or may not be used. For instance it's doubtful I'll ever use my GeForce GTX
>460's double-precision computing capabilities. But that's fine, it's relatively
>small and it's not worth designing a separate chip for the people who do use it.
>
>So I have nothing against dedicated hardware in general, but like I said it has
>to offer a high enough efficiency advantage, weighed against its utilization. The
>problem with some of the GPU's dedicated hardware is that even during it's key application,
>graphics, it's often either a significant bottleneck or >mostly idle.

You have said that, but frankly, you've said a lot of things that are simply wrong.

How about you provide some hard data on modern high performance GPUs (e.g. most recent generation from NV or AMD) on the utilization of the rasterizer. They have performance profilers, so it shouldn't be too hard. Then you can find out how much power the rasterizers use, and we can compare it to the power consumption of SW rendering. Then you will have actually a marginal understanding of the relative efficiency.

And I'm fairly certain that you will find that comparison to be very unattractive for SW rendering.

>Unifying vertex
>and pixel processing removed the bottleneck between them >and increased utilization.
>Texture sampling is useless to generic computing and >having too few texture units
>is a bottleneck to graphics, while the importance of FP32 >texture filtering increases,
>so it makes lots of sense to start doing the filtering in >shader units and have
>more generic gather/scatter units. And support for >micropolygons would require substantial
>hardware to sustain the peak throughput, but it's again >idle during other workloads
>and even for graphics it's full capacity isn't used all >the time. Make it smaller,
>and it's a bottleneck when drawing micropolygons. Again >unification seems like the better option here to me.

You haven't even quantified the gains from utilization at all for rendering, or the cost in terms of power consumption.

>>>What you're also forgetting is that the software evolves as well. In 2001 people
>>>were really excited about pixel shader 1.1. Today, a desktop GPU with only pixel
>>>shader 1.1 support would be totally ridiculous, regardless of how power efficient
>>>it is. I've said it before; we don't need more pixels, we >need more exciting ones.
>>>Which means increasing generic programmability.
>>
>>So let the shaders evolve, and stay separate.
>
>I sincerely hope you're not being serious. There's no way >GPU manufacturers will un-unify their architectures.

Please read what I wrote, carefully and think about it. "Stay separate" implies they are already separate. What are they separate from? You seem to assume I'm talking about the vertex/pixel/geo shaders being separate from one another, but that's hardly clear.

What was meant is that the shaders should stay separate from the CPU (which is the state today, even in IGPs).

>>Every single fact that I've seen tends to suggest that software rendering is a demonstrably bad idea.
>
>You haven't demonstrated anything.

Sure I have. CPUs are not optimized for throughput and have roughly 4X lower performance efficiency. In fact, in some cases that's a vast understatement.

A Tesla has roughly 2.2 GFLOP/s per W (DP). A high performance Westmere has roughly 0.75 GFLOP/s per W. Cayman is roughly 2.7 GFLOP/s per, although a real workstation card would be lower, probably around 2.5 GFLOP/s per W.

So the reality is that the performance per watt is much worse on CPUs than GPU, by a factor of 3-4. So to achieve the same throughput, the power consumption would be 3-4X higher. So...um...CPUs aren't throughput optimized.

>And "tends to suggest" coming from someone who's clearly basing things on prejudice
>is just more handwaving. I've proven you WRONG about the necessity for dedicated
>texture decompression, using real data.

You have no real data. You had bad data from an old simulator that the author of the simulator thought was BS. Garbage in, garbage out.

>And a 6-core CPU with FMA has 6 times more
>FLOPS than SwiftShader currently uses. Furthermore, I've shown that texel fetching
>is currently a huge bottleneck and gather/scatter would >put the CPU on par with
>the GPU's capabilities,

No you haven't. "On par with" means +/-20% to me, and I don't think a high-end CPU would come close.

>plus speed up many other graphics operations and other throughput-oriented
>applications. I've also shown that the battery life of a >laptop while gaming wouldn't
>be much lower than when using an IGP.

Actually your claims about power are probably the worst.

>And finally I've shown that an IGP does cost
>quite a bit and is worthless for non-graphics applications.

That you definitely haven't shown. And IGPs are useful for the same general purpose applications that a GPU is. Fusion parts will have OpenCL and compute shader. So will Ivy Bridge.

>Bandwidth, throughput, power efficiency, cost, everything is within reach to produce
>a more powerful CPU with adequate graphics capabilities to make the IGP redundant.
>And this whole discussion has made me even more confident >it won't stop there.

Congratulations, you have entered the twilight zone.

>>>As I've shown in my previous response, the IGP is bandwidth
>>>limited and software rendering is catching up with it. >Things are converging, and
>>>it will lead to much more exciting computer graphics (and >other appliations). So
>>>there's no need to fight it. Relatively things become less >efficient, but note that
>>>GT110 is far less efficient per transistor than a 40 nm >NV20 as well. In absolute
>>>terms the efficiency still improves thanks to >semiconductor advances. This is not about to stop.
>>
>>Yes and that is why GPU shaders are evolving. What you are suggesting is that
>>we throw away 10-15 years of GPU evolution and simply graft on some major features
>>to CPUs, and hope it works. Even Intel's approach with LRB was much more reasonable.
>
>Yes, GPUs are evolving too, toward a more CPU-like >architecture! I've proven that many times now.

Yes and the relative gap in performance is still HUGE.

>And why would you even care if this means throwing away 10-15 years of GPU evolution?
>Did you shed a tear when sound cards became redundant? >There's plenty of other examples
>of technology that didn't survive evolution.

No, because A) Creative Labs was a bunch of incompetent idiots and B) sound only took up 1-2% of my CPU cycles. Notice how that's not true of graphics.

>But GPU technology doesn't completely go to waste. AMD probably learned a thing
>or two from ATI on how to improve throughput efficiency (Bulldozer), and NVIDIA
>can produce ARM processors with well balanced IPC, DLP and >TLP.
>
>Heck, I wouldn't mind if CPU technology went to waste, if >it meant that GPUs are
>efficient enough at complex tasks to run Windows, and was >sitting in the center
>of my motherboard. I'll rename that it to "CPU" then >though.

Yeah, let me know when the GPU can actually compile stuff.

The bottom line is that while it's true that GPUs and CPUs are evolving towards one another, that says nothing about how vast the distance between the two is. The reality is that there is roughly a 4X gap in performance efficiency between GPUs and CPUs on many throughput workloads, and the gap is even larger on latency sensitive workloads.

Throughput means more than just scatter/gather although it is one key aspect. But to simply throughput down to scatter/gather is pure ignorance and naivete, and shows an acute lack of understanding of the substantial differences in circuit design, microarchitecture and software.

Anyway, unless you can really produce some real data, I'm done with this conversation.

DK