Hi David,

David Kanter (dkanter@realworldtech.com) on 2/7/11 wrote:
---------------------------
>Nicolas Capens (nicolas.capens@gmail.com) on 2/7/11 wrote:
>---------------------------
>>Hi David,
>>
>>David Kanter (dkanter@realworldtech.com) on 2/7/11 wrote:
>>---------------------------
>>>The fundamental point is that efficiency *really* does matter a lot. In a world
>>>where power is the #1 performance limiter, doing stuff in dedicated hardware becomes
>>>increasingly attractive. If you can cut the energy required to render a frame in
>>>half by using an IGP (which is quite believable), then you have gained a huge amount
>>>of energy to expend in the CPU or other areas.
>>>
>>>The key is understanding that with Moore's Law, spending more transistors to lower
>>>power is VERY attractive. That dictates more dedicated hardware, not less. Just
>>>look at what Intel has done - putting embedded micro-controllers inside CPUs to manage power.
>>>
>>>The whole idea of eliminating throughput oriented cores goes against that entirely.
>>
>>Who's talking about eliminating throughput oriented cores? I'm talking about adding
>>gather/scatter to CPU cores so they *become* efficient throughput oriented cores
>>and the functionality of the IGP can be unified.
>
>You are talking about eliminating throughput cores. You claim to have a basic
>understanding of hardware, so it should be readily apparent how CPU cores and throughput
>cores (e.g. Niagara, GPU shaders) differ.

Throughput-oriented is a term which originates from server systems, long before graphics chips were even called GPUs! It merely means a focus on data rate, potentially at the cost of latency. Any multiprocessor system, is a throughput oriented system. Clock frequency and ILP are latency-oriented, while DLP and TLP are throughput-oriented.

Today's x86 CPUs exploit DLP and TLP (SIMD, multi-core and Hyper-Threading) and have a less aggressive clocking than several years ago. So they are definitely throughput-oriented architectures already and would become more efficient at with the addition of gather/scatter support. Parallel load/store is the main thing setting them apart from GPUs.

GPUs are, in the words of NVIDIA's chief scientists, "aggressively throughput-oriented processors". Note though that GF104 features superscalar execution, intended to lower the latency. And aside from reducing bandwidth, caches also reduce latency. So GPUs are forces to become less aggressive at using throughput-oriented techniques, because reducing latency somewhat also reduces the amount of on-chip storage you need. It's a balancing act, because obviously reducing latency costs transistors as well.

>Even to someone without circuit design
>expertise, it should be blinding obvious - the clock speeds are about a factor of 2-4X different.

Then why did NVIDIA decide to put its aggressively throughput-oriented cores into a higher clock domain? The GeForce GTX 560 Ti has a shader clock of 1645 MHz, while the Radeon HD 6950 has a clock of 800 MHz. Does that mean NVIDIA's architecture is not throughput-oriented?

Clock speed alone isn't an indication of being throughput-oriented or not. It's a design decision which doesn't have to compromise *effective* throughput, as proven by NVIDIA vs. AMD. Another example is Cell BE, which clocks at 3.2 GHz but even at 90 nm was considered strongly thoughput-oriented. x86 CPUs have come a long way since the single-core Pentium 4 days, and they're not about to stop increasing throughput efficiency (AVX, Bulldozer, FMA, etc.).

As a matter of fact the CPU's clock frequency has remained nearly constant. The i7-2600K has a 3.8 GHz Turbo Mode ( but under multi-threaded workloads it's not that high), the same as the 2004 Pentium 4 Prescott. In the same time period, NVIDIA's GPUs have increased their clock frequency by over a factor 3. There's no indication this is going to change. For NVIDIA to conquer the HPC market, it needs to continue investing into latency reduction. To prevent an excessive growth in die size, it needs to increase the clock frequency. GF100 had some thermal issues, but they got that under control with GF110, which has more cores enabled and even higher clocks.

So while it's "blinding obvious" that there's a clock frequency difference today, it's also "blinding obvious" they're on a collision course. Gather/scatter support is still several years out, so by that time they'll have converged even closer and gather/scatter is the keystone.

>You cannot simply tack on scatter/gather to a latency optimized CPU core and expect
>it to look like a throughput core in terms of power efficiency. At least, there
>is definitely a lack of evidence for any such claims. Moreover, you need to preserve
>the power efficiency for workloads that cannot be vectorized.

An architecture which balances latency and theoretical throughput, can still achieve high effective thoughput. It's how NVIDIA achieved to outperform AMD with only half the FLOP density.

The way things converge, tacking on gather/scatter support does put the GPU within striking distance, starting with the IGP. For someone not playing games all the time, a balanced homogenous architecture is the most cost effective solution for all his processing needs.

Note that widening the vectors amortizes the cost of things like out-of-order execution. At the same time, AMD has reduced its VLIW width from 5 to 4, in order to achieve higher efficiency. GPUs also introduced concurrent kernel execution and scalar execution, and have growing register files and caches. So they're investing more transistors into latency reduction and programmability than raw FLOPS. GF110 has a 0.52 FLOP/transitor ratio. With G92b that was still a 0.94 ratio.

It's easy to see where your preconceptions come from though. NV40 had a 0.24 ratio, which G92b increased by a fourfold in a few years time. But you got fooled into thinking that this is a trend which can be sustained. Widening a component only increases the overall throughput desity of that component till it reaches 50% of the die area. And the components themselves get fatter to increase programmability as well, and the rest of the architecture needs to support the same throughput.

So for your own sake stop staring yourself blind at theoretical thoughput. There's a lot more to effective performance than that.

>>If dedicated hardware was the universal answer, then GPUs >would still have separate
>>vertex and pixel pipelines. So clearly you're not taking >all the of the factors into account!
>
>Newsflash: GPU shaders are dedicated hardware. The fact that the workloads for
>vertex and pixel shading are similar enough to use the same hardware is not relevant.

That's old news. Shader cores are no longer dedicated. NVIDIA now calls them CUDA cores instead, for good reason. The floating-point operations are IEEE-754 compliant, just like on a CPU. A Tesla card is as dedicated to graphics as it is to biochemical research. A CPU with gather/scatter would be just as dedicated to those. The only things making the GPU dedicated to gaphics, are the texture samplers, ROPs, and rasterizers (each of which are also optimized in a software renderer when there's gather/scatter support).

And please explain to me why the workload is not relevant to unification, instead of handwaving. If pixel shading still only consisted of integer operations, there wouldn't be any significant reason to unify them, and shader cores would be truely dedicated to graphics. Unification is strong proof that dedicated hardware is not the universal answer. If you still think it's irrelevant to the discussion, please elaborate.

>Let's take another example. Icera makes a very cool SDR. However, to meet the
>performance and power efficiency requirements, they use a custom designed chip to
>run the SDR. So, the 'dedicated hardware' is used by many different radio protocols,
>in exactly the same way that GPU shaders are used by many different shader types. It's still dedicated hardware though.

Does Icera's SDR support IEEE-754? I guess not, so *this* is irrelevant.

It's nothing personal, but face it, you're running out of arguments and start handwaving and reaching for absurd examples which I'm easily able to debunk.

>>Dedicated hardware leads to bottlenecks or idle silicon. >It's always either one
>>or the other. Both are a waste. So you need a big enough >performance difference
>>and high enough utilization to justify it.
>
>In case you haven't noticed, modern CPUs are filled with idle silicon. Floating
>point units, AES crypto blocks, virtualization support, real mode support, etc. Many of these were added recently.

Floating-point is useful to graphics, so this isn't an argument against software rendering.

As for AES, virtualization, real mode, etc. they certainly don't "fill" the CPU with idle silicon. Unless you can prove me otherwise, AES doesn't take die space proportional to the GPU's, ROPs, texture samplers or rasterizers. And like I said before, fast AES support is important for generic encrypted disk and network access, and gather/scatter speeds up software AES so the dedicated hardware can be removed. VT-x and real mode are even supported by Atom cores, so it's doubtful this takes any noticable die space on a desktop chip, and it's obviously indispensable for the software that make use of it.

Besides, like I said before GPUs also have lots of programmability features which may or may not be used. For instance it's doubtful I'll ever use my GeForce GTX 460's double-precision computing capabilities. But that's fine, it's relatively small and it's not worth designing a separate chip for the people who do use it.

So I have nothing against dedicated hardware in general, but like I said it has to offer a high enough efficiency advantage, weighed against its utilization. The problem with some of the GPU's dedicated hardware is that even during it's key application, graphics, it's often either a significant bottleneck or mostly idle. Unifying vertex and pixel processing removed the bottleneck between them and increased utilization. Texture sampling is useless to generic computing and having too few texture units is a bottleneck to graphics, while the importance of FP32 texture filtering increases, so it makes lots of sense to start doing the filtering in shader units and have more generic gather/scatter units. And support for micropolygons would require substantial hardware to sustain the peak throughput, but it's again idle during other workloads and even for graphics it's full capacity isn't used all the time. Make it smaller, and it's a bottleneck when drawing micropolygons. Again unification seems like the better option here to me.

>>What you're also forgetting is that the software evolves as well. In 2001 people
>>were really excited about pixel shader 1.1. Today, a desktop GPU with only pixel
>>shader 1.1 support would be totally ridiculous, regardless of how power efficient
>>it is. I've said it before; we don't need more pixels, we >need more exciting ones.
>>Which means increasing generic programmability.
>
>So let the shaders evolve, and stay separate.

I sincerely hope you're not being serious. There's no way GPU manufacturers will un-unify their architectures.

>>So there are opposing forces at play. Sometimes adding >dedicated hardware wins.
>>Sometimes adding generic hardware wins. So shed your >prejudices about software rendering
>>and look at the facts.
>
>Every single fact that I've seen tends to suggest that software rendering is a demonstrably bad idea.

You haven't demonstrated anything.

And "tends to suggest" coming from someone who's clearly basing things on prejudice is just more handwaving. I've proven you WRONG about the necessity for dedicated texture decompression, using real data. And a 6-core CPU with FMA has 6 times more FLOPS than SwiftShader currently uses. Furthermore, I've shown that texel fetching is currently a huge bottleneck and gather/scatter would put the CPU on par with the GPU's capabilities, plus speed up many other graphics operations and other throughput-oriented applications. I've also shown that the battery life of a laptop while gaming wouldn't be much lower than when using an IGP. And finally I've shown that an IGP does cost quite a bit and is worthless for non-graphics applications.

Bandwidth, throughput, power efficiency, cost, everything is within reach to produce a more powerful CPU with adequate graphics capabilities to make the IGP redundant. And this whole discussion has made me even more confident it won't stop there.

>>As I've shown in my previous response, the IGP is bandwidth
>>limited and software rendering is catching up with it. >Things are converging, and
>>it will lead to much more exciting computer graphics (and >other appliations). So
>>there's no need to fight it. Relatively things become less >efficient, but note that
>>GT110 is far less efficient per transistor than a 40 nm >NV20 as well. In absolute
>>terms the efficiency still improves thanks to >semiconductor advances. This is not about to stop.
>
>Yes and that is why GPU shaders are evolving. What you are suggesting is that
>we throw away 10-15 years of GPU evolution and simply graft on some major features
>to CPUs, and hope it works. Even Intel's approach with LRB was much more reasonable.

Yes, GPUs are evolving too, toward a more CPU-like architecture! I've proven that many times now.

And why would you even care if this means throwing away 10-15 years of GPU evolution? Did you shed a tear when sound cards became redundant? There's plenty of other examples of technology that didn't survive evolution.

But GPU technology doesn't completely go to waste. AMD probably learned a thing or two from ATI on how to improve throughput efficiency (Bulldozer), and NVIDIA can produce ARM processors with well balanced IPC, DLP and TLP.

Heck, I wouldn't mind if CPU technology went to waste, if it meant that GPUs are efficient enough at complex tasks to run Windows, and was sitting in the center of my motherboard. I'll rename that it to "CPU" then though.

Cheers,

Nicolas