Gabriele Svelto (gabriele.svelto@gmail.com) on 1/13/11 wrote:
>Honestly, I still have to see a system - outside of embedded applications - that
>doesn't have a GPU.

There would be no innovation whatsoever if an argument like this had any validity. Just because you haven't seen something yet doesn't mean it won't be viable in the future. Back in 2005 people had yet to see a consumer GPU with a unified vertex and pixel shader unit. That didn't stop it from becoming the dominant architecture really fast.

All it takes for CPU software rendering to become viable is an affordable mainstream CPU capable of delivering adequate graphics performance for the low-end market at acceptable power consumption. It may not be here yet but it's well within reach. Performance/Watt is doubling every two years, at any price point, and gather/scatter would get things ahead of that curve.

So no matter what the market's expectations are, at some point in the future the CPU will be capable of supporting it. A few years ago I've had people telling me software renderers would never run Oblivion or Crysis, unless at several seconds per frame on a high-end machine. Some even believed that it was somehow physically impossible to achieve the same image quality as a GPU. They were all proven wrong right the next year.

Sometimes I get the idea that the only impression people like you get when I speak the words "software rendering" is a memory of playing Quake at 320x200 at 8 FPS. There has been tremendous progress since those days! There is more tremendous progress coming! For the record, CPUs are hardware too, so software rendering is a bit of a misnomer. And shaders are software too. There are fewer architectural distinctions between CPUs and GPUs every generation.

>And no, if you go low in the CPU scale you lose a lot of performance.
>You have a high-clocked quad-core, how many of those end up in laptops? Cheap OEM
>PCs? CULV laptops? How fast would SwiftShader run 3DMark06 on a dual-core 1.4GHz
>CULV processor? Here's my guess: much slower than the pitiful IGPs you get with those machines.

Once again you're stuck in today. Instead try to look at the past and project that progress into the future. Back when they were launched, a Core 2 Duo at 2.93GHz costed 1000 USD. Nowadays you can get a superior Core i3 at 3.06 GHz for 115 USD, consuming only a fraction of the power. And if it didn't include an IGP it would have costed even less. In the AMD camp 100 USD will even buy you a quad-core already.

And with all due respect it's kind of lame that you have to use CULV systems as an example. It's pretty obvious they'll be the last to switch to software rendering. However, their low power consumption hasn't stopped them from using HD Audio codecs (despite high CPU usage), while as you pointed out earlier handheld devices have dedicated DSPs to lower the power consumption further. It indicates that they're just several years behind on the curve, but will surely follow.

>Well, Sandy Bridge IGP, which is a really nice surprise coming from Intel, scores
>over 4k in 3DMark06, and that's with early drivers:

Look at where it's heading. It's a 5x gap today but four years from now we could have gather/scatter and 8-core mainstream CPUs that don't break a sweat at 5 GHz. So that's roughly 3x the performance thanks to the full-featured AVX extensions and another 3x for the cores and clock frequency. Together with some fine-tuning of the software renderer we're looking at about 8000 3DMark06 points, on par with an 8800 GTS!

By then 3DMark06 will be showcased in musea next to dot-matrix printers though. The games of 2015 will require fully generically programmable GPUs that don't buckle when running long complex shaders. This requires technology found in CPU architectures... So you might as well use the actual CPU cores.

Let me put it this way: At a certain point the framerate of an old application no longer matters. Nobody really cares if something like Unreal Tournament 2003 is only slightly faster on a GF104 versus a G92, despite having nearly three times the transistors, since it's running in the range of 400+ FPS anyway. What actually matters is which of the latest games your hardware can handle adequately. As the programmability of graphics APIs went up the CPU became increasingly more efficient at implementing the graphics pipeline. As the ALU:TEX ratio of modern games continues to increase, the inefficiency at texture sampling starts to matter less and less. Gather/scatter is all that is needed to make the CPU excel at modern graphics.

>http://nl.hardware.info/reviews/1947/45/intel-core-i7-2600k-i5-2500k-i5-2300-sandy-bridge-review-gpu-benchmarks
>
>That's 10 times what you'd achieve with SwiftShader on those two cores.

You probably meant four cores? Or six if you replace the IGP's die space with CPU cores.

>And 3DMark06 is a basically CPU-less load, with many
>games you wouldn't have that luxury, StarCraft 2 for example which is pretty light
>for todays standards burns you 2-3 cores just for IA and physics.

StarCraft II is an interesting case because while it requires 3 cores or more to run optimally, the average CPU usage isnt't that high: http://www.pcper.com/article.php?aid=958. And given that 30 FPS is plenty for a strategy game this leaves even more CPU power unused and the game is GPU limited. So to overcome Amdahl's Law you could give the CPU some other tasks to work on as well. If you move some of the graphics workload from the GPU to the CPU you could have a cheaper GPU and still achieve the same performance. If this trend continues, sooner or later you're best off running all graphics on the CPU.

It also shows that games are not just about graphics. They make use of the generic capabilities of CPUs to deliver a more exciting experience than just pretty images. If you ditch the GPU and instead have a CPU with more cores and gather/scatter support it opens up a whole new world of possibilities. After all we don't really want any more pixels. We want pixels which show something interesting. Computing physics and AI is not something the GPU is succesful at. We're a long way from running the entire game on the GPU. For that to happen it would have to be fully generic like the CPU itself. But obviously that means it's more likely for the graphics to be moved to the CPU than for everything else to be moved to the GPU.

>So no, putting
>two additional cores on Sandy Bridge instead of an IGP would have been a terrible
>idea for 99% of the users. Having a decent IGP on the other hands enables them to
>play games, Oh, and removing the IGP would also means doing video decoding on the
>CPU which would have shortened battery life when you're on the move, etc...

That 99% is nonsense. Like I said before systems with X3100's are still sold by the millions today. Out of all the people who would buy a Sandy Bridge system without dedicated graphics, there has to be a large percentage who would be happy with lesser graphics if it made things cheaper or the CPU itself faster.

>Honestly, a 6-core SB using software rendering wouldn't be able to play games with
>entry-level settings

The Crysis benchmark runs at 17 FPS on my Core i7 920 @ 3.2 GHz. On a 6-core Sandy Bridge it would be running at over 30 FPS. Still without making use of AVX (let alone gather/scatter).

>You seem to have completely missed the point: AMD GPUs use VLIW processors. VLIW
>has been since its inception a design methodology for processors that trades off
>complexity for higher maximum throughput. VLIW processors will have lower utilization
>*by design* compared to a RISC approach, but they will make up for it by offering
>higher maximum throughput and compute density because they are simpler and thus
>smaller to implement. The low utilization of AMD GPU's is a *design choice* which
>actually allowed them to hit the same performance targets with a smaller die.

I don't see how VLIW is less complex than SIMD. It needs to route multiple control words to each unit, it needs support for horizontal operations, and it requires a shuffle network.

But by keeping the clock frequency lower, AMD can push TSMC's 40 nm process to the integration density limit. They can have fewer pipeline stages, route the wires closer together with less fear of interference, and use smaller transistor gate widths. They also don't need buffers between the clock domains. All this was necessary to compensate for the VLIW complexity and lower utilization. But on top of that they also decided on smaller caches, fewer ROPs and less bandwidth. This causes AMD's architectures to frequently become bottlenecked, further lowering utilization. So let's not confuse these two.

At first it might seem they succeeded at making a smaller chip perform the same, but it's not without compromises. Fermi is still more feature-rich and won't stall as easily. So if you take that into account they've created an equivalent at best. There's no such thing as a free lunch.

So to get back to the actual point: NVIDIA proves that you can do with less GFLOPS, if you use them efficiently.

>>And since there are lots of GPGPU applications which
>>don't even achieve 10% of the theoretical performance of the GPU, while they use
>>the CPU to it's fullest, it's clear that CPUs are really efficient at juggling tasks
>>around and keeping the data flowing.
>
>We already knew that, but that doesn't change a thing for graphics.

It does change things for graphics, because graphics becomes ever more irregular. NVIDIA's architectures have ever lower computing density (per transistor) but keep up with AMD's architectures. Amdahl's Law, a.k.a. the law of diminishing returns, makes AMD's approach less scalable.

>>So how AMD and NVIDIA compare today is pretty irrelevant to the discussion.
>
>You were the one to bring this argument to the topic, not me.

Please read the context. I literally meant "today". If all you care about is today's games, AMD's current architecture is as good a choice as NVIDIA's. The relevant part is how things will evolve in the future.

>>What
>>matters in the long term is that applications, including graphics applications,
>>are getting more complex and a CPU architecture is more suited for this.
>
>Then why GPUs are becoming more and more prevalent? First it was only desktop computers,
>now it's desktop and laptops, now you get them even in DVRs, cellphones, tablets,
>you name it. If they are on-die or not doesn't change a basic fact: if you want
>to do graphics at decent performance level and in a decent power envelope you need
>dedicated hardware. Complex, programmable dedicated hardware but still dedicated.

Dedicated sound processing came and went. Dedicated physics processing came and went. Dedicated RAID controllers are also no longer a must and hybrid RAID and sofware RAID is taking its place...

Dedicated graphics has a much longer cycle because it's more complex and requires more performance for adequate support, but the end is getting nearer. Every stage of the graphics pipeline is becoming programmable. Programmability is the enemy of dedicated.

There's actually a physical reason why everything that is programmable, eventually moves to the CPU. Every time the semiconductor feature size halves, you can have twice the number of wires, but up to four times the transistor density. What this means is that eventually everything becomes bandwidth limited. On-die memory (registers and caches) delay the issue but it's starting to matter less and less what's doing the actual processing. This makes a single processor which is capable of all tasks, a lot more interesting than a heterogenous design which can't balance the workload.

>>Note that software rendering frees game engine developers from the graphics API restrictions, which in turn results in higher performance in practice: http://graphics.cs.williams.edu/archive/SweeneyHPG2009/TimHPG2009.pdf
>
>I had already read that presentation and while Tim Sweeny has already predicted
>the demise of GPUs many times GPUs have been entering new markets where they were
>inexistent in the past. So no, this is a purely theoretical argument that doesn't
>only happen in practice: it's also being proven wrong by the fact that we're getting
>*more* dedicated hardware, not less.

Entering new markets doesn't mean they have been succesful at it yet. A 2500 USD Tesla card (which has been tested extensively to avoid the 2-10% failure rate of the consumer market) might sound like a fair deal to the HPC market but it's not what Sweeney had in mind. The average GPU in a typical desktop sytem isn't capable of running anything additional to the graphics workload, while achieving better performance than when making use of the CPU's idle cores instead.

And I've already given you examples of dedicated hardware which became irrelevant because the CPU became powerful enough to take over their role. New dedicated hardware may come, but if it benefits from programmability it will eventually go.

>>Retirement buffers are pretty small. But instead of speculative execution, GPUs
>>opt for massive simultaneous multi-threading, which means they need register space
>>for all these extra threads. Shaders which use more registers than what the hardware
>>was designed for, can really decimate performance. Developers also want a true call
>>stack, so you need massive caches to store all this context.
>>This situation is not sustainable. Some really drastic measures need to be taken
>>to reduce the thread count.
>> But you don't need to look any further than CPU architectures.
>>Speculative execution, branch prediction, forwarding, register renaming, etc. can
>>come to the rescue.
>
>GPU caches are smaller than CPU caches, how's that comes? GPU register files are
>pretty large, but very dense, do you think that a ROB is a small/cheap structure?
>Or a branch predictor history table? Or OoO logic? They're not and for throughput-oriented
>workloads they do not make much sense. How comes thread count on CPUs is going up?

Register files are less dense than caches. But anyway, the real problem for GPUs is the number of threads (strands).

Why does Fermi still not support recursion? Because it requires many kilobytes of stack space, per thread. On the CPU, 1 MB of stack space per thread is typical. Even in the worst case, that's still likely to fit in L3 cache. Even without recursion the call stack can get pretty deep for complex code. With thousands of threads in flight, a GPU just can't have the on-die storage (regardless of whether it's registers or slightly denser caches).

So to ever support recursion or deep call stacks, GPUs will need to drastically reduce their thread count. You need the tricks used in CPUs to achieve that.

>Besides you are the first person I hear saying that current GPU trends are not sustainable,
>if it came from a GPU HW designer it might make sense but from you it sounds like wishful thinking.

NVIDIA's GPU designers have added superscalar execution. This complicates the scheduler and actually creates the risk that they can't find two independent operations in the shader. So clearly some benefit outweighed these disadvantages...

This is GPU designers telling me that threads need to be processed faster in order to avoid running out of register space.

That said, why would it matter who's saying something, for it to make sense? I don't want GPUs to not support recursion. It's just a technical fact. If they did support it, I wouldn't be having this discussion and I would write software for the GPU. And for your information, I have a masters degree in computer science and engineering, with a minor in embedded systems.

If you have a better technical explanation why NVIDIA went superscalar, please let me hear it. No qualifications necessary.

>> But when GPUs sacrifice computing density for higher efficiency
>>at complex generic tasks, it's obviously also a really interesting option to just
>>start with a multi-core CPU architecture and give it powerful graphics capabilities by adding gather/scatter.
>
>Yes, they tried that, it didn't work out.

No, they tried that in the high-end market and it didn't work out. They created an architecture that was ahead of its time. Larrabee does support recursion, for one thing, but no software uses it because no GPU supports it. It's a chicken-and-egg problem. You need Larrabee on the market for developers to make good use of its capabilities, but you can't sell Larrabee when it's up against high-end GPUs running legacy benchmarks.

Approaching things from the low-end market first does work, because it's not necessary for every dollar of hardware to be worth its maximum in graphics performance. It also buys you a CPU. So while Larrabee is only judged by its ability to render graphics, a CPU is valuable for lots of other things. So as long as it's reasonably efficient at graphics, there's bound to be a market for it. And gather/scatter will make that final difference in efficiency which makes a GPU redundant.

>>While texture sampling is a highly specialized operation (it's the only thing Larrabee
>>really has dedicated hardware for), the general trend is still to generalize them
>>into load/store (gather/scatter) operations.
>
>No, the general trend is to have more, more powerful texture units in GPUs. Look
>for yourself at the latest architecture from *any* graphics vendor if you don't believe me.

Sure, in absolute numbers there are more texture units each generation, but the TEX:ALU ratio has been going down. A smaller percentage of die space is going to texture units. But shaders and compute kernels also access unfiltered local and global data. For this you need load/store units.

>>There are several reasons for this:
>>
>>Shaders need ever fewer texture samples (TEX:ALU ratio). So the amount of die area
>>use on texture samplers has steadily decreased. But this means that TEX heavy shaders
>>become bottlenecked, and for ALU heavy shaders they are underutilized...
>>
>>But while the TEX:ALU ratio decreases, the shaders do make more unfiltered memory
>>accesses. So modern GPUs also have specialized access to local memory, shared memory,
>>global memory, etc. Each of these can again be a bottleneck!
>
>Actually they're no bottleneck at all, they are part of the specialized hardware
>that makes GPUs so fast at graphics. They have tremendous bandwidth, they are optimized
>for texture access patterns and they are usually fully associative. No matter how
>you slice it, TEX:ALU ratio has decreased and still TUs have increased in number
>and functionality. They are not going away, they are getting better and they're
>both faster and more efficient at doing their job as a CPU will ever be. On top of that they use less space on the die.

The early graphics chips were pretty much only texture units and raster operation units. You needed multiple passes to do anything interesting. Nowadays the texture units occupy just a fraction of the die space. And you can easily get bottlenecked by texture accesses.

The lower TEX:ALU ratio has helped software rendering to catch up. Arithmetic operations map almost directly to CPU instructions. Texture sampling still takes many instructions but since they're not so frequent any more the CPU has relatively speaking become better at graphics. Just compare running 3DMark2001 and 3DMark06 using SwiftShader. The drop in 3DMark points is smaller than when using a GPU.

Anyway, that doesn't take away that gather/scatter support would still make a critical difference in texture sampling efficiency for the CPU, and it would also helps speed up other parts of the graphics pipeline.

>>Also, filtering is useless for GPGPU, while graphics on the other hand wants full
>>FP32 filtering (possibly even FP64). These diverging needs are hard to combine into
>>texture units or other highly specialized load/store units.
>
>All recent GPUs can do full-speed FP32 filtering, and filtering is very important
>for GPGPU too because many GPGPU applications are actually doing graphics and so
>they can also benefit from texture hardware.

No they can't: http://www.anandtech.com/show/4008/nvidias-geforce-gtx-580/2

>>So it becomes very tempting to just slap all these different forms of memory access
>>together, have generic load/store units, and let the cache hierarchy take care of local and temporal coherence.
>
>Tempting for who? All hardware available now or in the design pipe is going in
>the other direction, how do you fail to see this basic, hard fact?

What other direction? The latest GPUs now have the ability to gather any of the four components of a 2x2 footprint and query the texture LOD, from within the shader! They're also performing part of the addressing in the shader units. So it's already really close to being fully programmable.

The fact of the matter is that there's a massive number of shader units and they're frequently underutilized. So why have dedicated hardware for addressing and filtering? By slimming down the texture units they can have more of them, and by unifying them with load/store units the bottlenecks are reduced.

I'll respond to the rest of your visionless arguments some other day. Got a SIGGRAPH submission deadline coming up...