Hi Davhid,

>Nicholas,
>
>Due to the length, I need to trim a fair number of comments, especially those related
>to SW rasterization techniques (and respond in a separate post).

Fair enough.

>>>David Kanter (dkanter@realworldtech.com) on 1/27/11 wrote:
>>>---------------------------
>>>That shows nothing about compression, that merely tells about the change in performance
>>>due to larger textures. It's also largely about an older game that isn't designed for 2560x1600.
>>
>>The change in performance is the whole point.
>
>No it's not. You made a specific claim about texture compression. Comparing large
>vs. small textures says NOTHING about absolute importance of compression, but only of relative compression.

I'm sorry but I'm still having a hard time understanding what you're trying to say here. What exactly do you mean by large vs. small textures? Just to be clear again; at Ultra High settings, Doom 3 uses the exact same textures, only uncompressed. They're the same dimensions, just bigger in storage size. So how is that saying nothing about absolute importance of compression? It's exactly the situation for a software renderer: It uncompresses the texture entirely so it can be sampled like any other texture.

Doom 3 is the only game I currently know of which allows testing the absolute difference in performance when using uncompressed textures of the same dimension. But while it's old, it should be more texture bandwidth limited than today's games with low TEX:ALU ratio so I still think this result is very relevant to my claim: "Hardware support for compressed textures is not critical to the viability of software rendering."

>>Doom 3's uncompressed textures are equal in dimensions to >the compressed ones.
>
>I didn't say dimensions, I said size and bandwidth. Again, you claimed that texture
>compression does not save meaningful amounts of bandwidth. I see no proof of that claim.

No, I claimed it only helps by about 10%. And I was obviously talking about absolute performance since the previous sentence was the claim about viability.

And it only helps that little simply because (a) applications are not bottlenecked by bandwidth all the time, and (b) a signficant portion of the bandwidth is also use for other purposes.

>>And I've tested 3DMark06 with SwiftShader
>3DMark06 is not reflective of modern games. It's 5 years old now!

May I ask what your expectation is for modern games if a stress test like 3DMark06 doesn't show any sign of being bandwidth limited? Do you honestly expect a sudden difference in texturing bandwidth so significant it makes this result meaningless?

>> while forcing >the mipmap LOD down one
>>level (which is equivalent to reducing the texture bandwidth by a factor 4), and
>>the SM3.0 score went from 250 to 249. Yes, due to some >statistical variance the
>>score was actually lower. If texture bandwidth was of >great significance, you'd expect a much higher score.
>Yes, but by reducing the texture size you have screwed up the compute:bandwidth
>ratio. Compressing textures will always improve that ratio...simply using smaller textures will preserve that ratio.
>
>By using smaller textures you changed the workload substantially.

I didn't touch the compute workload at all. I merely made it sample from a smaller mipmap level, reducing the bandwidth per sample footprint by a factor of 4, which is about the same as hardware would benefit from compressed textures.

>>>You're claiming that for #2 the difference is 10% and I don't see any real evidence
>>>of that. Compression should be vastly more effective.
>>
>>Texture compression rates of 1:2 and 1:4 are very common, >but that doesn't translate
>>into big performance improvements. Most of the time >there's sufficient bandwidth
>>headroom to allow uncompressed textures without an impact >on performance.
>
>And what about power? If I can transfer 2X or 4X less bytes, I can use a smaller (or slower) memory interface.

You can't use a smaller (or slower) memory interface without affecting all other applications. So that's just not an option. The memory subsystem is what it is so you might as well make use of it. It has ample bandwidth and large caches, which compensate for the lack of dedicated texture decompression hardware.

Also, while the additional transfers indeed take some power, dedicated texture decompression hardware takes power too.

>>And even
>>in bandwidth limited situations, there's already a large >chunk of it used by color,
>>z and geometry. So the performance won't drop by much.
>
>I don't believe any numbers you've provided on this. The simulator results you
>posted were very old and the author of the simulator basically indicated that they weren't valid or useful.
>
>[snip]
>
>>>I happen to know the author of that study in question. The data is INCREDIBLY
>>>OLD. It's from a simulator that did not use any Z or color compression, so the results cannot be taken seriously.
>>
>>Yes, it's from a simulator called ATTILA. And for the >record it did use z compression to generate that graph.
>
>I am familiar with the tool and with the author. The data in question came from
>an old version of Attila, and as per my discussion with Victor Moya (the author)...the data is pretty much useless.

Ok, then please show me recent data you think is more relevant. Regardless of the age of the data I used I think it's very reasonable to conclude that applications aren't bandwidth limited all the time and a significant portion of the bandwidth is used by data other than texels. It's up to you to disprove this now.

>>>I think you underestimate the cost of adding pins to your >>memory controller.
>>
>>I'm not suggesting adding extra pins to make software >rendering viable. It's already viable bandwidth-wise.
>
>You're suggesting getting rid of texture compression. That will increase bandwidth usage.

Yes it will, but it's insignficant since often there's some bandwidth headroom left, and other data consumes bandwidth too so the absolute increase in bandwidth is far less than the 1:2 or 1:4 texture compression ratio. And finally the large caches are compensating for it as well.

>>>Actually for the mobile space it does have to come close to dedicated hardware. Battery life matters, a lot.
>>
>>For laptops we see the graphics solution range from IGPs >to lower clocked high-end
>>desktop GPUs. So while battery life is important, it doesn't mean the TDP has to
>>be the lowest possible. It just has to be acceptable. A cheaper IGP which consumes
>>more power is likely to sell better than a more expensive >efficient one.
>
>Power consumption != TDP.
>
>And I strongly suspect that even a cheap IGP is going to be more efficient (power-wise)
>than software rendering on the CPU.

Quite possibly. And indeed the battery life depends on the average power consumption and not TDP. It is likely to include a lot of idle time or simple things like scrolling though a web page. But note again that not so long ago this was all handled adequately by the CPU and DMA operations. Nowadays CPUs have a lot better performance per Watt, so I'm not worried about battery life during light work.

>>Also note
>>that today's GPUs have far more features than the average >consumer will really use,
>>meaning they are less energy efficient than they could >have been. But the TDP is
>>still acceptable for a good battery life.
>
>No offense, but you don't seem to understand the difference between TDP and dynamic
>power consumption. They are only loosely related. Both are important, but I suspect
>SW rendering falls flat on its face for the latter.

There can be some difference in power consumption, but it's not an order of magnitude or so. And my point was that even a GPU consumes more power than what is strictly necessary for the task, but that's acceptable when you get other things in return. TDP is the wrong word to use in this context since it only determines the worst case battery life and not the average battery life, but what I meant is that a somewhat higher power consumption doesn't mean it's not viable. I've owned laptops with a battery life of two hours and ten hours when idle. You may think I unquestionably prefer the latter, but the two hour one was much cheaper and more powerful.

>>>>Most likely the bandwidth will just steadily keep increasing, helping all (high
>>>>bandwidth) applications equally. DDR3 is standard now accross the entire market,
>>>>and it's evolving toward higher frequencies and lower voltage. Next up is DDR4,
>>>>and if necessary the number of memory lanes can be >increased.
>>>
>>>More memory lanes substantially increases cost, which is >>something that everyone wants to avoid.
>>
>>They'll only avoid it till it's the cheapest solution. >Just like dual-channel and
>>DDR3 became standard after some time, things are still >evolving toward higher bandwidth
>>technologies. Five years ago the bandwidth achieved by >today's budget CPUs was unthinkable.
>
>Actually it was quite predictable. We're still using the same basic memory architecture - dual channel DDRx.

No it wasn't quite predictable. Even in late 2007 it wasn't clear what the benefit of dual-channel was: http://www.tomshardware.com/reviews/PARALLEL-PROCESSING,1705-11.html

It's the de facto standard now, even for mobile and budget systems, but it still took some steady evolution to get to this point. Nowadays triple channel is pretty high-end and quad channel is coming too, but in due time it will be affordable for more market segments. 256-bit memory busses have been pretty common for graphics cards for many years now. So while extra pins are expensive they're not prohibitively expensive, when the need arrives.

Anyway, we're nowhere near that point yet. And by that time the caches will be even more massive so it's unlikely that graphics will require anything in addition to the steady evolution.

>>>>And again CPU technology is not at a standstill. With >T-RAM just around the corner
>>>>we're looking at 20+ MB of cache for mainstream CPUs in >the not too distant future.
>>>
>>>T-RAM is not just around the corner.
>>
>>This news item suggests otherwise: http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20090518005181
>
>Please stop making me repeat myself. How about we make a gentleman's wager?
>
>You're apparently confident that TRAM will be shipping in products soon. I'm confident it won't.
>
>So let's define what soon means, and then we can step back and see who's right and who is wrong.
>
>For me, soon means a year.

The news sites say T-RAM is being prepared for GlobalFoundries's 32 and 22 nm nodes. Since it's unlikely to be used by Bulldozer, it will probably slip to the 22 nm revision.

So my soon means two years.

>>But even if it does take longer, it doesn't really matter to the long-term viability
>>of software rendering. There will be a breakthrough at some point and it will advance
>>the convergence by making dedicated decompression hardware totally unnecessary (if
>>it even has any relevance left today).
>
>It totally matters. If you are expecting a magical 2-4X increase in cache density
>that is iso-process, then you might as well just give up. And yes, it seems to
>me that much of what you are claiming is predicated on a magical increase in cache density.

It's not a necessity. Before the end of the decated a cache size of 256 MB will be perfectly common even without any technology breakthrough. Of course a breakthrough would help to rapidly expand the viability of software rendering to other markets, but it doesn't predicate the viability for the initial market.

>>>>And while the expectations for 'adequate' graphics go up as well, it's only a slow
>>>>moving target. First we saw the appearance of IGPs as an adequate solution for a
>>>>large portion of the market, and now things are evolving >in favor of software rendering.
>>>
>>>I think if you look at the improvement in IGPs, that's a >>very FAST improving target.
>>
>>The hardware isn't the target.
>
>Yes it is. To be competitive, your solution needs to have comparable power and
>adequate performance to a hardware solution.

It depends on your definition of comparable. Looking at the demise of dedicated audio processing again, it's clear that some difference in power efficiency is acceptable. The cost reduction is well worth it.

For argument's sake, let's say the successor to Sandy Bridge is identical except for the fact that the IGP consumes only one tenth of the power, at a 5$ cost increase. Does that mean nobody will buy Sandy Bridge any more? Of course not.

So again, the power consumption of other solutions is not the target. If the power consumption is within the consumer's expectation and it costs less, it will sell.

>>>Swiftshader is swiftshader - other SW rendering systems >work differently. They
>may (or may not) see similar >benefits.
>>
>>Again, why does that make SwiftShader's results only "meaningful" for SwiftShader?
>
>That's easy. Because other SW renders may work *differently* from swiftshader.
>If they work *differently*, they will have *different* performance and *different*
>bottlenecks and *different* performance gains from the changes in Sandy Bridge.
>
>Do you see a theme in the above statement?

No offence, but I only see handwaving. Exactly how do you imagine another software rendreer to work differently, in a way that makes the 30% performance increase at 55% bandwidth not meaningful?

Note that the shaders have a fixed number of arithmetic operations and data accesses. There's not all that much you could do "differently" to execute them as fast as possible without wasting cycles or bandwidth.

What you're saying is equivalent to saying that the results for one video codec implementation are meaningless to other implementations of the same codec or similar codecs. Yes they'll differ, but unless they perform completely differently they'll have closely resembling characteristics and the benchmark results of one implementation on a new CPU will be meaningful to the other implementations as well.

Anyway, I'd love to proven wrong and learn about a totally different software rendering approach...

>Let me give you a hypothetical example here. Say GCC runs 20% faster on Sandy
>Bridge than Nehalem. Now what does that tell you about the performance gain for LLVM or MSVC on Sandy Bridge?

It tells me that LLVM and MSVC will see similar performance increases. Each of them manipulates small data elements (string characters), does lots of pointer chasing, contains lots of branches, etc. If you're compiling the same source code and applying the same optimizations, the performance gain on Sandy Bridge will be comparable. If not, it means one of them is not using an optimal implementation in the first place. After switching to a better implementation, the performance gain will be more in line with the other compilers.

Just look at the problem space. You have similar input and the same CPU architecture to begin with. You can't have two optimal implementations with vastly different characteristics.

In particular for the case of software rendering I do not know of any different but still efficient approach for which SwiftShader's results are meaningless.

>>Anyway, to meet you in the middle I downclocked my >i7-920's memory from 1600 MHz
>>to 960 MHz, and the 3DMark06 SM3.0 score went from 250 to >247. So once again, reducing
>>the bandwidth to 60% has no significant impact on >performance.
>
>Alright...now we are getting somewhere, thanks for taking the effort to look into
>this! So you definitely have shown that for 3DMark06, there is not a big bandwidth bottleneck.
>
>However, I care about modern games. Could you try and run something like Crysis or Civ 5 or 3D Mark Vantage?

Crysis at 1680x1050, 4xMSAA, High detail, 64-bit: 0.89 FPS versus 0.85 FPS at 60% bandwidth.

Now, before you say this is evidence that bandwidth is sligtly more critical with modern games, I've also tested without AA and got 1.23 FPS versus 1.22 FPS. So clearly it would be more useful to focus on color compression than texture compression.

Also, these framerates are obviously unbearable but I've also tested my laptop's Quadro FX 380M and only got 0.77 FPS. And even at low detail playing Crysis on a previous generation IGP doesn't seem like a great experience: http://www.youtube.com/watch?v=qwrNP0CgPa8

Note once again that SwiftShader isn't using AVX yet, let alone gather/scatter...

>>>Reducing the number of loads and stores isn't really relevant. It's the number
>>>of operations that matters. If you are gathering 16 different addresses, you are
>>>really doing 16 different load operations.
>>
>>Not with Larrabee's implementation. It only takes as many >uops as the number of
>>cache lines that are needed to collect all elements, per >load unit.
>
>Yes, that's exactly what I said. It's the access pattern that's a problem.

It's not. Texture sampling can use a blocking technique (also called tiling or address swizzling) to ensure that texels have a high locality in both directions. Lookup tables, ROP operations, vertex attribute fetch, interpolant setup, etc. it all has a high locality.

Since Sandy Bridge already has two load units, the worst case for gathering eight 32-bit elements would be a mere 4 cycles, which is already way better than the extract/insert emulation. The typical throughput will be much closer to 1 every clock cycle though.

>>>No, that's totally different. With unified shaders you can easily use more or
>>>less geometry...until you run out of physical shaders to execute on. You should
>>>look at Kayvon's work on micro-polygon rendering.
>>
>>Easily? I think you're seriously underestimating the >complexity of adapting your
>>software to the hardware. Checking whether you're vertex >or pixel processing limited
>>wasn't feasible in actual games ten years ago, and it >still isn't.
>
>Sure you can, there are profiling tools for that. Nvidia and ATI both have them.

That only gives you global results. Different shaders can still be heavily bottlenecked by different resources. For instance a post filter effect is likely limited by texel fetch while procedural shaders or complex SH lighting is likely compute limited. There's only so much the developer can do about that.

A better hardware architecture is to get rid of the dedicated texel addressing units and filtering. This frees up space for more shader units and more generic load/store units, lessening the bottlenecks for most workloads. Only a very well balanced shader could be slightly worse off due to taking some computing resources for filtering. But that's the exception. This is very similar to the unification of the vertex and pixel pipelines. The average workload is better off with a unified architecture.

Note that some GPUs already use the shader units for part of the texel address calculations. And they might perform FP32 filtering in the shader units as well. So mark my works: the days of dedicated texture sampling are numbered.

>>>>It's clear that telling software developers what (not) to >do doesn't result in
>>>>a succesful next generation hardware architecture. With >non-unified architectures,
>>>>there were numerous applications which were vertex >processing limited, and numerous
>>>>ones which were pixel processing limited. And even those >in the middle have a fluctuating workload.
>>>
>>>Yes, except a unified shader architecture doesn't really preclude that many options.
>>
>>That's what I'm saying.
>
>Um no. You said that programmable shaders are too limited, and you want programmable rasterization and texturing.

Yes, but making these things programmable is equivalent to unification, as I've shown above.

Texturing is close to becoming programmable/unified. Alpha blending is also a perfect candidate for being computed in the shader cores. Note that DX10 already made the alpha test stage programmable.

Rasterization is a bigger challenge so it will probably take longer to become programmable, but it's already actively being researched and the results are "encouraging": http://research.microsoft.com/en-us/um/people/cloop/eg2010.pdf

>I am not convinced that those two should be programmable, and I'm not convinced
>that FF hardware is really that restrictive.

With all due respect, that's because you lack vision of what would be possible with a fully programmable architecture. Don't get me wrong, I'm not claiming I know everything it would be used for. The only thing I'm sure of is that it would be pretty exciting. There's a lot of GPGPU research which remains unsuccesful for mainstream hardware, and there's lots of talk about ray-tracing, Reyes, micropolygons, and many other rendering techniques. A lot of new techniques even still need to see the light of day. But we first need the flexible hardware to enable them.

If you're still not convinced, please recall that John Carmack was practically ridiculed for wanting floating-point shaders. Most people were not convinced it would be necessary and that integer operations are too restrictive. It's pretty obvious nowadays that these people were wrong. And while back then Carmack only envisioned Shader Model 2.0, we're now at Shader Model 5.0 and developers continue to push the limits of the hardware's capabilities. OpenCL is only in its infancy.

>>>First, you do need to count the increases in resolution. Pixels have definitely
>>>increased over time.
>>
>>Yes the resolution has increased but everything else scaled accordingly. More pixels
>>doesn't mean higher benfit from texture compression. In fact TEX:ALU is going down,
>>meaning pixel shaders are more compute limited than >bandwidth limited.
>
>I'm skeptical. Available bandwidth always grows more slowly than compute...

Which is fine. As the computational complexity of software increases, it uses more temporary data. The cache hierarchy can keep up with the core's bandwidth needs to store temporary results. End results got to RAM memory. But since the resolution only increases slowly the current steady evolution in RAM bandwidth suffices.

Note that before pixel shaders, color operations were essentially performed by the ROP units, and you needed multiple passes to create various effects. Shaders store the intermediate results locally, in the registers. The bandwidth needs went down (but obviously it then allowed other things to consume the freed up bandiwdth). Nowadays, the GPU doesn't just have massive register files, it also has caches. They're still tiny in comparison to the register files and the computing power, but they'll need to grow to accommodate the working sets of increasingly more complex software. The new data is uncompressed, so there's no way around using caches.

For CPUs the cache size per FLOP is going down. CPUs and GPUs are converging on every front. Of course if it increases again due to a breakthrough in density that's obviously not a bad thing. So let me rephrase that as the die area per FLOP spent on cache is converging.

>>>>In ten more years caches could be around 256 MB, and >that's without taking revolutionary
>>>>new technologies like T-RAM into account. So it's really >hard to imagine that this
>>>>won't suffice to compensate for the texture bandwidth >needs of the low-end graphics market.
>>>
>>>Because you are imagining that the low-end market stays put. It won't.
>>
>>I didn't say it stays put. I said it's a slow moving target. Evidence of this is
>>the ever growing gap between high-end and low-end graphics hardware. IGPs were born
>>out of the demand for really cheap but adequate 3D graphics support. They cover the majority of the market:
>>http://unity3d.com/webplayer/hwstats/pages/web-2011Q1-gfxvendor.html
>>
>>This massive market must obviously have a further division >in price and performance
>>expectations. Some people want a more powerful CPU for the >same price by sacrificing
>>a bit of graphics performance, while others simply want a >cheaper system that isn't
>>aimed at serious gaming. As the CPU performance continues >to increase exponentially,
>>and things like gather/scatter can make a drastic >difference in graphics efficiency,
>>software rendering can satisfy more and more people's expectations, even if those
>>expectations themselves incease slowly.
>
>In essence, what you are saying is that some people would be fine with lower performance
>graphics. That's something I agree with.
>
>I just don't know what the relative performance of SW rendering is to dedicated
>hardware, and how that curve will change over time.

It started to converge ever since the end of the MHz race and they started to focus on performance per Watt. GPUs are essentially limited by performance per Watt as well. But they're pretty much out of architectural options to increase it. They have to rely on new semiconductor processes. CPUs benefit from that equally, but at the same time they started to keep the clock frequency moderate, increased the number of cores and increased the vector width. This has massively increased the performance per Watt, beyond what the advancement in process technology offers. And they still have FMA up their sleeves.

As for translating this into software rendering performance, it's unfortunately held back by the lack of gather/scatter. But I've shown you plenty of data now which shows that the IGP is going to go the way of the dodo.

>My sense is that hardware is probably getting relatively faster, given the attention Intel is paying.

You have to take into account how much of that is due to increasing the area and/or power consumption. It's easy to mistake a performance increase as an architectural advancement. For several years GPUs were able to increase their ALU density dramatically, but now they're hitting walls. Instead their best hope now is to increase the utilization of these ALUs for complex workloads, using unification and extending the memory hierarchy.

Back to Intel's IGPs, their increased performance is due to additional Execution Units and higher clock frequency. It means that from a cost and power consumption point of view they're not improving relative to software rendering.

>>>That's because it won't.
>>
>>It will. The only strengths the GPU has left are all >components based on the ability
>>to load/store lots of data in parallel. The CPU cores >already achieve higher GFLOPS
>>than the IGP
>
>That's true today, but I suspect it won't be true in the future.

Do you have any indication to back up this suspicion? CPUs still have FMA to come, and I see no indication that the CPU core count is stagnating. Ever more applications see some benefit from quad-core, and Bulldozer will default to eight cores. Haswell is claimed to default to eight (beefier) cores as well.

Note that the non-K models of Sandy Bridge have half the EUs. So the average IGP has a lot of catching up to do to exceed the CPU's GFLOPS. Against an 8-core Haswell CPU it would have to become 8 times more powerful to exceed it. Fat chance.

It's far more interesting to give the CPU cores gather/scatter support and ditch that sorry IGP.

>Also remember
>that you have to share those FLOP/s with other tasks.

If you replace the IGP with CPU cores you get about the same amount of FLOPs in return. Furthermore, with an IGP every game is GPU limited. As unification proves, sharing FLOPs is not a bad thing. Even if the other tasks need lots of FLOPs, it's typically only temporary. For instance to compute physics. At other times the CPU is sitting idle waiting for the IGP to finish.

By unifying the whole thing the bottlenecks are gone and you get closer interaction. Currently a lot of GPGPU applications are unsuccesful because of the round-trip latency. You need to send a massive amount of data-parallel workload to the GPU to compensate for that. That doesn't always work out so well. A lot of applications want to process small amounts of work instead and get the results fast to interact with it.

Software rendering allows to break free from the legacy graphics pipeline and perform only exactly the operations you want. For instance when doing a post filter effect you don't need perspective correction or mipmap LOD calculations. You can already do this with a compute shader, but then again a compute shader IS software rendering...

>>You
>>can either ditch the IGP to make things cheaper, or >replace it with additional CPU
>>cores so you get a really powerful processor for any >workload.
>
>I think to achieve IGP level of performance, using an IGP is the most efficient in terms of power and area.

Yes, but you have to look a the total picture. The IGP is only used intensively during gaming. At other times its transistors are largely a waste of die space (money). Having more CPU performance/dollar can be of greater importance to the consumer. So just like the most cost effective solution for audio processing is a software codec, we're close to the point where for some markets an IGP-less system offers the best balance.

>>Quoting t-ram.com: "T-RAM Semiconductor has successfully developed the Thyristor-RAM
>>technology from concept to production-readiness. Our Thyristor-RAM technology has
>>been successfully implemented on both Bulk and SOI CMOS. "
>>
>>Sounds like production ready to me.
>
>Not even close.

What makes you so sure? Yes, Z-RAM appears to be a failure but I'd expect them to think twice before making statements like jointly developing it for application in 32 and 22 nm nodes. Since the work on Bulldozer started many years ago and they probably didn't want to take any risks it's highly unlikely to use 32 nm T-RAM, but it could appear in a 22 nm refresh in about two years time.

So do you have any other information aside from the irrelevant comparison to Z-RAM to claim T-RAM is not even close to production?

>>>>2nd gen Z-RAM
>>>
>>>Doesn't work at all.
>>
>>Maybe not as cache memory, but it's hopeful as a DRAM replacement: http://www.z-ram.com/en/pdf/Z-RAM_LV_and_bulk_PR_Final_for_press.pdf
>>
>
>Just a moment ago, you were suggesting it as a cache replacement. Now you suddenly
>are back-tracking? And nobody really wants a proprietary DRAM replacement.

Yes, I wasn't aware that 2nd gen. Z-RAM is not considered as an SRAM replacement. Either way the point was that cache technology is not at a standstill and one less candidate for improving the density doesn't put a lid on it.

>>>It's possible, but they will need to become more competitive from an energy perspective with fixed function stuff.
>>
>>There's not a lot of fixed-function stuff left. The >majority of the GPU's die space
>>consist of programmable or generic components.
>>
>>And I've shown before that the CPUs FLOPS/Watt is in the >same league as GPUs:
>>- Core i7-282QM: 150 GFLOPS / 45 Watt (more with Turbo Boost)
>>- GeForce GT 420: 134.4 GFLOPS / 50 Watt
>
>The GT 420 is ancient. A better comparison would be the GT 440, which is 96 shaders,
>1.6GHz and 65W. That's ~300 GFLOP/s for 65W, or a roughly 2X advantage.

Ancient? Both the GT 420 and 440 use a GF108 chip.

And my calculator says it's only a 50% advantage. But that's without taking into account that Sandy Bridge's TDP includes the IGP and includes Turbo Mode. So that advantage likely goes up in smoke.

FMA will double the GFLOP/s score for a minimal increase in transistor count. Add to this powerhouse gather/scatter support and NVIDIA has a serious problem.

>>Obviously software rendering requires a bit more >arithmetic power to implement
>>the remaining fixed-function functionality, but >programmable shaders take the bulk.
>>
>>So there's no lack of energy efficiency. The CPU simply >can't utilize its computing power effectively
>
>GPUs are definitely more power efficient than CPUs.

At effective graphics performance, yes, but that's easy enough to fix with FMA, integer AVX, and gather/scatter.

Advances in power gating technology ensures that CPUs don't waste power on things which are idle. Also the Physical Register File of Sandy Bridge singificantly reduces the power consumption of the out-of-order execution logic.

So today's CPUs are pretty lean and mean and they're only going to get better. But they don't have to match the effective power efficiency of GPUs to make software rendering viable.

>>All applications that contain loops can benefit from >gather/scatter. That's all applications.
>
>If that's true, then what % performance increase could we expect to see in SPECint?

A 15% reduction in memory accesses and 28% of scalar instructions converted into vector instructions: http://personals.ac.upc.edu/mpajuelo/papers/ISCA02.pdf. This paper uses a dynamic technique, but with a bit of assistance from the programmer (like the use of the 'restrict' keyword) the same results can be achieved statically.

Without gather/scatter a large number of loops can't be vectorized: http://hpc.cs.tsinghua.edu.cn/research/cluster/SPEC2006Characterization/auto_para.html

>>With sather/scatter support every scalar operation would >have a parallel equivalent.
>>So any loop with independent iterations can be >parallelized and execute up to 8 times faster.
>
>That's assuming there is no control flow divergence.

Simple flow control is still worth vectorizing.

>>And I don't think the hardware cost is that high. All you >need is a bit of logic
>>to check which elements are located in the same cache >line, and four byte shift
>>units per 128-bit load units instead of one, to collect >the individual elements.
>>Note that logic for sequentially accessing the cache lines >is already largely in
>>place to support load operations which straddle a cache >line boundary.
>
>You are saying that because you don't design hardware. What you are suggesting
>is in fact, quite complicated and large.

I don't currently design hardware but I have a masters degree in computer engineering, with a minor in embedded systems. I've read 'Digital Integrated Circuits - A Design Perspective' by Rabaey et al. front-to-back so by all means please elaborate on just how complicated and large it would be.

Also please tell me how Larrabee can have 512-bit wide gather/scatter support for each of its tiny cores while a pair of 128-bit gather/scatter units would be quite complicated and large.

Unless of course by quite complicated and large you meant about as complicated and large as texel fetching logic. Sure, it's definitely not trivial to design and the area is not insignificant. But it seems well worth it given that it will allow the vectorization of code which previously wasn't vectorizable.

>>>Really? Have you heard of Vertica? They do an awful lot >>of lossless compression of data in memory.
>>
>>No, I hadn't heard about them before. Could you point me >to some document where
>>they detail how they added hardware support for compressed >memory transfers to reduce bandwidth?
>
>They don't need hardware to do lossless compression. They have a clever column
>oriented database. Check vertica.com. One of their big performance gains is from reducing memory (and disk) bandwidth.

You were previously talking about the importance of dedicated texture decompression hardware. Now you're telling me about Vertica and how they don't need hardware to do lossless compression...

I don't see the relevance of Vertica to this discussion, unless you're actually trying to say dedicated hardware isn't that important after all.

Indeed there are also lossy and lossless techniques to reduce memory bandwidth which can be implemented in software. That includes textures. It's currently not worth the cycles though, as software rendering isn't bandwidth limited.

>>>Many applications use adjacent values.
>>
>>Yes, and many applications also use non-adjacent values.
>>
>>If a loop contains just one load or store at an address >which isn't consecutive,
>>it can't be vectorized (unless you want to resort to >serially extracting/inserting
>>addresses and values). So even if the majority of values >are adjacent, it doesn't
>>take a lot of non-adjacent data to cripple the performance.
>
>You can still vectorize it, you just need to have a bunch of scalar loads/stores
>to deal with the non-adjacent addresses.

That's exactly what I said (note the "unless"). You don't generally want to do that though. Note that you need two instructions (extract and insert) to emulate a single scalar load. So you risk making things slower than the scalar code. See slide 44 here: http://sc.tamu.edu/help/softwareDocs/intel/tutorial/compiler_1.pdf

It would help to have an instruction which takes a vector element as address offset (e.g. "mov ymm0.3, dword ptr [rax+ymm1.3]"), but to really tackle Amdahl's Law we need gather/scatter support which in the ideal case takes a single cycle.

>>>>Why? It only accesses the cache lines it needs. If all >elements are from the same
>>>>cache line, it's as fast as accessing a single element.
>>>
>>>And exactly as fast as using AVX! i.e. no improvement >>and more complexity/power.
>>
>>No. The addresses are unknown at compile time. So the only >>option with AVX1 is
>>to sequentially extract each address from the address >vector, and insert the read
>>element into the result vector. This takes 18 instructions.
>
>>With gather support it would be just one instruction. >Assuming it gets split into
>>two 128-bit gather uops, the maximum throughput is 1 every >cycle and the minimal throughput is 1 every 4 cycles.
>
>>>>But even in the worst case
>>>>it can't generate more misses or consume more bandwidth.
>>>
>>>It sure can. Now instead of having 1-2 TLB accesses per cycle, you get 16. How
>>>many TLB copies do you want? How many misses in flight do you want to support?
>>
>>You're still not getting it. It only accesses one cache >line per cycle. It simply
>>has to check which elements are within the same cache >line, and perform a single
>>TLB access for all of these elements. Checking whether the >addresses land on the
>>same cache line doesn't require full translation of each >address.
>
>That's quite complicated hardware, and you can't afford to have that on the critical
>path for any of your normal loads. So now you need a fairly separate load/store pipeline for scatter/gather.

I don't think any singnificant additions are needed on the critical path itself. It just requires four byte shift units intead of one, but they operate in parallel as well. Computing which elements go where can be done up front, before entering the critial path for normal loads. It would be perfectly acceptable to have a higher latency for scatter/gather, if necessary.

It's definitely an engineering challenge, but so far I can't think of anything which would jeopardize the feasibility.

>>Nothing other than graphics runs better on the IGP. As >I've mentioned before, GPGPU
>>is only succesful using high-end hardware.
>
>Today...unclear what tomorrow holds.

He who predicts the present is always right.

Anyway, yes, GPUs are highly likely to become better at GPGPU applications. But it will require improving the efficiency for executing workloads which differ from graphics. Which means fewer graphics-specific fixed-funciton hardware, more programmability, more unification, superscalar scheduling, concurrent kernels, larger caches, etc. Everything is pointing in the direction of the GPU becoming more CPU-like. Which means at some point it makes no sense at all to keep things heterogeneous.

>>So the CPU is better than the IGP at absolutely everything >else. That makes it
>>really tempting to have a closer look at what it would >take to make it adequately efficient at graphics as well.
>>
>>The answer: gather/scatter.
>
>It would also need a 2X improvement in FLOP/w and /mm2, possibly more.

Adding FMA support and tossing out the IGP roughly triple the compute density. Performance per Watt is already excellent as proven by the comparison against the GeForce 420 and 440.

>>Multi-core, 256-bit vectors, Hyper-Threading, software >pipelining... the CPU is
>>already a throughput device! It's just being held back by >the lack of parallel load/store
>>support. It's the one missing part to let all those GFLOPS >come to full fruition.
>
>You keep on repeating this as if it were true, but it's not. I agree that lack
>of scatter/gather is an issue. But a more fundamental issue is that throughput
>optimized cores (e.g. shader arrays) are simply more efficient for compute rich
>workloads. You can't really get around that.

You keep on repeating that throughput optimized cores are "simply" more efficient. I've given you dozens of detailed arguments why despite that, software rendering is the future, while you're just handwaving based on the prejudice that CPUs are weak and power hungry.

Yes GPUs are throughput optimized so evidently they are more efficient for compute rich workloads, but they fall flat on their face when running out of registers or if the working set doesn't fit in the cache or if the code is too divergent or if the work batches are too small, etc. CPUs cope much more gracefully with increasing software complexity.

So it's getting less relevant just how efficient GPUs are at compute rich workloads. Nobody cares if they can run Max Payne at 1000 FPS. What matters in the long run is the newer workloads, which are less data parallel and less coherent.

Now, GPUs obviously still dictate the pace at which application developers diverge from compute rich workloads. So we're not going to see for example ray-traced games tomorrow. But GPUs do still suck at these kind of workloads and no amount of additional shader cores is going to help it. They'll need to evolve in the direction of CPU architectures to enable new workloads.

It also means that CPUs don't have to become as compute optimized as today's GPUs for software rendering to take over. Although they'll still drastically improve at it with FMA and gather/scatter, they have plenty of other valuable features to become the dominant architecture for any workload.

>>What specialized hardware would that be? I've already shown that texture compression
>>hardly makes a difference,
>
>No, you cited extremely old data from a simulator, where even the author of the
>simulator thinks the data is not useful.

No, I only gave that simulator data to clarify the results from actual experiments. It doesn't have to be very accurate to be useful. Regardless of what the exact bandwidth usage looks like today, there will be plenty of headroom and it's not just consumed by texturing.

If you want to debunk that, I suggest you show me recent data for which this isn't true, or tell me exactly why the author thinks the old data isn't useful and how it affects the validity of your dedicated texture decompression hardware importance claim.

>>and sampling and filtering is becoming programmable anyway.
>>Gather/scatter speeds up just about every other pipeline >stage as well.
>
>Except it doesn't benefit many workloads, and it costs a lot of area and power.
>So you want to disable it on the many workloads where it does not help.

>>>I totally agree that scatter/gather is a great capability to have. But what's
>>>the cost in die area, power and complexity? Not just to the core, but also the memory controller, etc.
>>
>>Larrabee has wider vectors and smaller cores, but features gather/scatter support.
>>So I don't think it takes a lot of die space either way. It doesn't require any
>>changes to the memory controller, just the load/store units. I'm not entirely sure
>>but collecting four elements from a cache line can probably largely make use of
>>the existing network to extract one (unaligned) value. And checking which addresses
>>land on the same cache line is a very simple equality test >of the upper bits.
>
>I think you have no or minimal experience designing hardware, so I'm not really
>inclined to take your word for it...especially compared against the expertise of
>the thousands of CPU designers at places like Intel, AMD and IBM.

Let me get this straight... You assume I have no knowledge of hardware design, without pointing out any flaw in my reasoning, and you're more inclined to turn towards experienced CPU designers such as those working at Intel, who added gather/scatter to Larrabee, as an indication why gather/scatter for the CPU isn't feasible?

>Scatter/gather is expensive and that's why it isn't done.

All things weren't done the day before they were done. You can't conclude from that that gather/scatter is (too) expensive.

Lots of things don't happen simply out of poor judgement. For instance some of the SSE instructions are just late fixups of old incomplete extensions. It doesn't mean they were expensive to add the first time around.

Gather/scatter is a significant deviation from the well-known scalar load/store unit. It's very alien for CPU designers and it requires considerable R&D even if the end result isn't necessarily expensive. Also, CPU designers are often clueless about the software applications. They just benchmark current software and try to come up with the next idea on how to execute it faster. But without gather/scatter support, lots of loops are not vectorized or the software developers compute things very differently. For instance computing an exponential function with scalar code is best done using some lookup tables, but with vector code you currently need to resort to long polynomials. This is then wrongfully interpreted as a need for more arithmetic performance. Another example is converting AoS data into SoA data for SIMD processing. This currently requires lots of shuffle operations between registers so CPU designers are inclined to add more and faster shuffle units. But with scatter/gather there wouldn't be any need to shuffle data accross registers.

Cheers,

Nicolas