Nicholas,

Due to the length, I need to trim a fair number of comments, especially those related to SW rasterization techniques (and respond in a separate post).

>David Kanter (dkanter@realworldtech.com) on 1/27/11 wrote:
>---------------------------
>>>Doom 3 on a GTX 460 at 2560x1600 4xAA runs at 53 FPS at >Ultra High Detail, and
>>>at 56 FPS at High Detail. I was being generous when I said >10%.
>>
>>That shows nothing about compression, that merely tells about the change in performance
>>due to larger textures. It's also largely about an older game that isn't designed for 2560x1600.
>
>The change in performance is the whole point.

No it's not. You made a specific claim about texture compression. Comparing large vs. small textures says NOTHING about absolute importance of compression, but only of relative compression.

>Doom 3's uncompressed textures are equal in dimensions to >the compressed ones.

I didn't say dimensions, I said size and bandwidth. Again, you claimed that texture compression does not save meaningful amounts of bandwidth. I see no proof of that claim.

>>What I'd want to see is for a number of MODERN games:
>>
>>1. Texture size (uncompressed vs. compressed)
>>2. Bandwidth usage (uncompressed vs. compressed)
>
>Texture size is irrelevant. They can be kept in compressed >form when unused. Only
>a fraction of all the texture levels is needed during a >frame.

>And I've tested 3DMark06 with SwiftShader

3DMark06 is not reflective of modern games. It's 5 years old now!

> while forcing >the mipmap LOD down one
>level (which is equivalent to reducing the texture bandwidth by a factor 4), and
>the SM3.0 score went from 250 to 249. Yes, due to some >statistical variance the
>score was actually lower. If texture bandwidth was of >great significance, you'd expect a much higher score.

Yes, but by reducing the texture size you have screwed up the compute:bandwidth ratio. Compressing textures will always improve that ratio...simply using smaller textures will preserve that ratio.

By using smaller textures you changed the workload substantially.

>>You're claiming that for #2 the difference is 10% and I don't see any real evidence
>>of that. Compression should be vastly more effective.
>
>Texture compression rates of 1:2 and 1:4 are very common, >but that doesn't translate
>into big performance improvements. Most of the time >there's sufficient bandwidth
>headroom to allow uncompressed textures without an impact >on performance.

And what about power? If I can transfer 2X or 4X less bytes, I can use a smaller (or slower) memory interface.

>And even
>in bandwidth limited situations, there's already a large >chunk of it used by color,
>z and geometry. So the performance won't drop by much.

I don't believe any numbers you've provided on this. The simulator results you posted were very old and the author of the simulator basically indicated that they weren't valid or useful.

[snip]

>>I happen to know the author of that study in question. The data is INCREDIBLY
>>OLD. It's from a simulator that did not use any Z or color compression, so the results cannot be taken seriously.
>
>Yes, it's from a simulator called ATTILA. And for the >record it did use z compression to generate that graph.

I am familiar with the tool and with the author. The data in question came from an old version of Attila, and as per my discussion with Victor Moya (the author)...the data is pretty much useless.

>And even though it's old, this data is still very >relevant. UT2004 has a very high
>TEX:ALU ratio, meaning that contemporary games are far >less likely to become bandwidth
>limited. UT2004 also has simple geometry and no pre-z >pass, again meaning that texture
>bandwidth has only become less relevant with newer games.

I'd have to see a fair number of modern games to know that.

[snip]

>>I think you underestimate the cost of adding pins to your >>memory controller.
>
>I'm not suggesting adding extra pins to make software >rendering viable. It's already viable bandwidth-wise.

You're suggesting getting rid of texture compression. That will increase bandwidth usage.

>Multi-core is driving the bandwidth needs up for all >applications, so I'm confident
>that it will be increased anyway in due time. But there's >no need for additional
>dedicated hardware. The caches do an excellent job at >reducing the overall bandwidth needs.

On modern games, with large textures, etc.?

[snip]

>>Actually for the mobile space it does have to come close to dedicated hardware. Battery life matters, a lot.
>
>For laptops we see the graphics solution range from IGPs >to lower clocked high-end
>desktop GPUs. So while battery life is important, it doesn't mean the TDP has to
>be the lowest possible. It just has to be acceptable. A cheaper IGP which consumes
>more power is likely to sell better than a more expensive >efficient one.

Power consumption != TDP.

And I strongly suspect that even a cheap IGP is going to be more efficient (power-wise) than software rendering on the CPU.

>Also note
>that today's GPUs have far more features than the average >consumer will really use,
>meaning they are less energy efficient than they could >have been. But the TDP is
>still acceptable for a good battery life.

No offense, but you don't seem to understand the difference between TDP and dynamic power consumption. They are only loosely related. Both are important, but I suspect SW rendering falls flat on its face for the latter.

>Furthermore, nobody expects a long battery life during >intense gaming. Even with
>dedicated graphics hardware the power consumption during >gaming is relatively high.
>So instead of a multi-core CPU with an IGP you might as >well have a CPU with a couple
>more cores. As long as the TDP is the same, it's >acceptable.

No it's not. Again, you don't seem to understand the difference between TDP and dynamic power consumption.

>>>Most likely the bandwidth will just steadily keep increasing, helping all (high
>>>bandwidth) applications equally. DDR3 is standard now accross the entire market,
>>>and it's evolving toward higher frequencies and lower voltage. Next up is DDR4,
>>>and if necessary the number of memory lanes can be >increased.
>>
>>More memory lanes substantially increases cost, which is >>something that everyone wants to avoid.
>
>They'll only avoid it till it's the cheapest solution. >Just like dual-channel and
>DDR3 became standard after some time, things are still >evolving toward higher bandwidth
>technologies. Five years ago the bandwidth achieved by >today's budget CPUs was unthinkable.

Actually it was quite predictable. We're still using the same basic memory architecture - dual channel DDRx.

>So frankly I don't care how they'll do it in the future, >but CPUs reaching 100's
>of GB/s of bandwidth will sooner or later be perfectly >normal.

You had better care, because it could easily drive up costs. The more you spend on your I/O the less area and power you have for compute.

>>>And again CPU technology is not at a standstill. With >T-RAM just around the corner
>>>we're looking at 20+ MB of cache for mainstream CPUs in >the not too distant future.
>>
>>T-RAM is not just around the corner.
>
>This news item suggests otherwise: http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20090518005181

Please stop making me repeat myself. How about we make a gentleman's wager?

You're apparently confident that TRAM will be shipping in products soon. I'm confident it won't.

So let's define what soon means, and then we can step back and see who's right and who is wrong.

For me, soon means a year.

>But even if it does take longer, it doesn't really matter to the long-term viability
>of software rendering. There will be a breakthrough at some point and it will advance
>the convergence by making dedicated decompression hardware totally unnecessary (if
>it even has any relevance left today).

It totally matters. If you are expecting a magical 2-4X increase in cache density that is iso-process, then you might as well just give up. And yes, it seems to me that much of what you are claiming is predicated on a magical increase in cache density.

>>>And while the expectations for 'adequate' graphics go up as well, it's only a slow
>>>moving target. First we saw the appearance of IGPs as an adequate solution for a
>>>large portion of the market, and now things are evolving >in favor of software rendering.
>>
>>I think if you look at the improvement in IGPs, that's a >>very FAST improving target.
>
>The hardware isn't the target.

Yes it is. To be competitive, your solution needs to have comparable power and adequate performance to a hardware solution.

>Consumer expectation is the target. Sandy Bridge
>leaves a hole in the market for consumers who want a >powerful CPU but are content with minimal graphics support.

How many consumers want that? 10? 100? 1K? 1M?


>>Swiftshader is swiftshader - other SW rendering systems >work differently. They may (or may not) see similar >benefits.
>
>Again, why does that make SwiftShader's results only "meaningful" for SwiftShader?

That's easy. Because other SW renders may work *differently* from swiftshader. If they work *differently*, they will have *different* performance and *different* bottlenecks and *different* performance gains from the changes in Sandy Bridge.

Do you see a theme in the above statement?

>All reviews only include a select number of benchmark >applications. Does that mean
>the results are meaningless for other applications? Of >course not.

It means that you need to look at a large number of samples (i.e. benchmarks) to ascertain overall performance.

Let me give you a hypothetical example here. Say GCC runs 20% faster on Sandy Bridge than Nehalem. Now what does that tell you about the performance gain for LLVM or MSVC on Sandy Bridge?

>Unless you can give me any sort of indication how another >software renderer with
>Shader Model 3.0 support could be bandwidth limited, 30% >higher performance with
>55% of the bandwidth is extremely meaningful for any such >software renderer.
>
>You seem to be suggesting that SwiftShader is doing something wrong which makes
>it 30% faster with 55% of the bandwidth. If that's the >case, great! It means that things can get much faster >still.

I'm saying it may work differently from other SW rendering approaches, and therefore have different bottlenecks.

[snip]

>Anyway, to meet you in the middle I downclocked my >i7-920's memory from 1600 MHz
>to 960 MHz, and the 3DMark06 SM3.0 score went from 250 to >247. So once again, reducing
>the bandwidth to 60% has no significant impact on >performance.

Alright...now we are getting somewhere, thanks for taking the effort to look into this! So you definitely have shown that for 3DMark06, there is not a big bandwidth bottleneck.

However, I care about modern games. Could you try and run something like Crysis or Civ 5 or 3D Mark Vantage?

>>Reducing the number of loads and stores isn't really relevant. It's the number
>>of operations that matters. If you are gathering 16 different addresses, you are
>>really doing 16 different load operations.
>
>Not with Larrabee's implementation. It only takes as many >uops as the number of
>cache lines that are needed to collect all elements, per >load unit.

Yes, that's exactly what I said. It's the access pattern that's a problem.

[snip]

>>No, that's totally different. With unified shaders you can easily use more or
>>less geometry...until you run out of physical shaders to execute on. You should
>>look at Kayvon's work on micro-polygon rendering.
>
>Easily? I think you're seriously underestimating the >complexity of adapting your
>software to the hardware. Checking whether you're vertex >or pixel processing limited
>wasn't feasible in actual games ten years ago, and it >still isn't.

Sure you can, there are profiling tools for that. Nvidia and ATI both have them.

[snip]

>>>It's clear that telling software developers what (not) to >do doesn't result in
>>>a succesful next generation hardware architecture. With >non-unified architectures,
>>>there were numerous applications which were vertex >processing limited, and numerous
>>>ones which were pixel processing limited. And even those >in the middle have a fluctuating workload.
>>
>>Yes, except a unified shader architecture doesn't really preclude that many options.
>
>That's what I'm saying.

Um no. You said that programmable shaders are too limited, and you want programmable rasterization and texturing.

I am not convinced that those two should be programmable, and I'm not convinced that FF hardware is really that restrictive.

Moreover, even if it is limiting...it's not obvious to me when you want to make a shift from FF to SW. Except that the next 2-3 years don't seem promising.

[snip]

>>First, you do need to count the increases in resolution. Pixels have definitely
>>increased over time.
>
>Yes the resolution has increased but everything else scaled accordingly. More pixels
>doesn't mean higher benfit from texture compression. In fact TEX:ALU is going down,
>meaning pixel shaders are more compute limited than >bandwidth limited.

I'm skeptical. Available bandwidth always grows more slowly than compute...

>>>In ten more years caches could be around 256 MB, and >that's without taking revolutionary
>>>new technologies like T-RAM into account. So it's really >hard to imagine that this
>>>won't suffice to compensate for the texture bandwidth >needs of the low-end graphics market.
>>
>>Because you are imagining that the low-end market stays put. It won't.
>
>I didn't say it stays put. I said it's a slow moving target. Evidence of this is
>the ever growing gap between high-end and low-end graphics hardware. IGPs were born
>out of the demand for really cheap but adequate 3D graphics support. They cover the majority of the market:
>http://unity3d.com/webplayer/hwstats/pages/web-2011Q1-gfxvendor.html
>
>This massive market must obviously have a further division >in price and performance
>expectations. Some people want a more powerful CPU for the >same price by sacrificing
>a bit of graphics performance, while others simply want a >cheaper system that isn't
>aimed at serious gaming. As the CPU performance continues >to increase exponentially,
>and things like gather/scatter can make a drastic >difference in graphics efficiency,
>software rendering can satisfy more and more people's expectations, even if those
>expectations themselves incease slowly.

In essence, what you are saying is that some people would be fine with lower performance graphics. That's something I agree with.

I just don't know what the relative performance of SW rendering is to dedicated hardware, and how that curve will change over time.

My sense is that hardware is probably getting relatively faster, given the attention Intel is paying.

>>>SwiftShader 1.0 was first used by a 2D casual game called Galapago. Despite the
>>>game's graphical simplicity, it was totally texture sampling limited and it was
>>>barely reaching the necessary 10 FPS for playability. That >was five years ago.
>>Today we have Crysis running at 20+ >FPS.
>>
>>What resolution and quality settings?
>
>800x600 at low detail. It's twice as fast as Microsoft WARP: http://msdn.microsoft.com/en-us/library/dd285359.aspx

Yes, but that's not a realistic resolution. Most displays are going to be probably 1280x960 (or something of that ilk).

>>That's because it won't.
>
>It will. The only strengths the GPU has left are all >components based on the ability
>to load/store lots of data in parallel. The CPU cores >already achieve higher GFLOPS
>than the IGP

That's true today, but I suspect it won't be true in the future. Also remember that you have to share those FLOP/s with other tasks.

>so gather/scatter unlocks that power for >graphics >applications.


>You
>can either ditch the IGP to make things cheaper, or >replace it with additional CPU
>cores so you get a really powerful processor for any >workload.

I think to achieve IGP level of performance, using an IGP is the most efficient in terms of power and area.

>>>http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20090518005181
>>
>>That means nothing. It means that GF is investigating the technology, not that it's production ready.
>
>Quoting the announcement: "...into a joint DEVELOPMENT >agreement targeted toward
>the APPLICATION of T-RAM’s Thyristor-RAM embedded >memory..."
>
>Emphasis mine. Why would a major foundry enter into a >development agreement with
>a startup, unless the technology has already been proven >on a smaller scale?

To *jointly investigate*. Look AMD even licensed ZRAM's first generation crap:
http://www.eetimes.com/electronics-news/4057964/AMD-licenses-Innovative-Silicon-s-SOI-memory

When did it get commercialized? Never...

>Quoting http://www.t-ram.com/news/media/3B.1_IRPS2009_Salling.pdf:
>"Taken together, the results of this study show
>that T-RAM is a reliable and manufacturable memory
>technology."
>
>Quoting t-ram.com: "T-RAM Semiconductor has successfully developed the Thyristor-RAM
>technology from concept to production-readiness. Our Thyristor-RAM technology has
>been successfully implemented on both Bulk and SOI CMOS. "
>
>Sounds like production ready to me.

Not even close.

>>>Anyway, there are multiple high density cache technologies. There's Thyristor-RAM,
>>
>>That's T-RAM.
>
>I know. I was just summing up "high density cache >technologies".
>
>>>1T-RAM,
>>
>>Not a replacement for SRAM.
>
>Why not? Looks useful as L3 cache to me.

>>>2nd gen Z-RAM
>>
>>Doesn't work at all.
>
>Maybe not as cache memory, but it's hopeful as a DRAM replacement: http://www.z-ram.com/en/pdf/Z-RAM_LV_and_bulk_PR_Final_for_press.pdf

Just a moment ago, you were suggesting it as a cache replacement. Now you suddenly are back-tracking? And nobody really wants a proprietary DRAM replacement.

My point is that all this stuff is totally unproven from a high volume perspective. It may be 'feasible', but that doesn't make it 'economical'. And in the case of ZRAM, it wasn't even feasible in the first place. TRAM seems better, but it's very unclear.

>>It's possible, but they will need to become more competitive from an energy perspective with fixed function stuff.
>
>There's not a lot of fixed-function stuff left. The >majority of the GPU's die space
>consist of programmable or generic components.
>
>And I've shown before that the CPUs FLOPS/Watt is in the >same league as GPUs:
>- Core i7-282QM: 150 GFLOPS / 45 Watt (more with Turbo Boost)
>- GeForce GT 420: 134.4 GFLOPS / 50 Watt

The GT 420 is ancient. A better comparison would be the GT 440, which is 96 shaders, 1.6GHz and 65W. That's ~300 GFLOP/s for 65W, or a roughly 2X advantage.

>Obviously software rendering requires a bit more >arithmetic power to implement
>the remaining fixed-function functionality, but >programmable shaders take the bulk.
>
>So there's no lack of energy efficiency. The CPU simply >can't utilize its computing power effectively

GPUs are definitely more power efficient than CPUs.


>During the early days of AC'97 there was some pretty >serious debate about moving
>the audio processing workload to the CPU. It made a real >difference in benchmark
>results. People who back then swore by the efficiency of >dedicated sound cards, now happily use HD Audio.

>>Perhaps when graphics
>>gets to that point, it will be fine to put it in SW.
>
>Exactly. There's no doubt it will happen, some day. My >take is that gather/scatter
>support is sufficient to initiate the move to software >rendering.

I bet HD Audio uses about 1-2% of the overall CPU cycles. Perhaps when graphics gets in that neighborhood - say 5-10% of the CPU cycles...it might be feasible. But I don't think that's really feasible today or in the near future.

>>And yes, power efficiency matters a lot. You may not >>think so, but it does.
>
>I do think it matters, a lot. But I think you're >underestimating how power efficient
>CPUs already are. It just doesn't translate into high >effective performance for
>3D graphics due to wasting a lot of cycles on moving data >elements around.

CPUs are simply not as power efficient as GPUs, because they spend more power on reducing latency of operations. There's pretty hard data to prove it as well. Consider the efficiency for ATI GPUs vs. a CPU.

>An AVX FMA instruction can perform 16 operations every >single cycle, but it would
>take a whopping 72 uops if every address and element was >extracted/inserted sequentially.
>When it comes to load/store, we haven't evolved beyond x87 >yet. Of course this is
>the worst case and typically not every vector load/store >has to be a gather/scatter,
>but for situations where you do need them it makes a >massive difference.

That's because the hardware for scatter/gather is expensive and power hungry.

[snip]

>>I don't think you design CPUs for high volume applications. Most don't need scatter/gather
>>and the hardware cost is high.
>
>All applications that contain loops can benefit from >gather/scatter. That's all applications.

If that's true, then what % performance increase could we expect to see in SPECint?

>With sather/scatter support every scalar operation would >have a parallel equivalent.
>So any loop with independent iterations can be >parallelized and execute up to 8 times faster.

That's assuming there is no control flow divergence.

>And I don't think the hardware cost is that high. All you >need is a bit of logic
>to check which elements are located in the same cache >line, and four byte shift
>units per 128-bit load units instead of one, to collect >the individual elements.
>Note that logic for sequentially accessing the cache lines >is already largely in
>place to support load operations which straddle a cache >line boundary.

You are saying that because you don't design hardware. What you are suggesting is in fact, quite complicated and large.


>>Really? Have you heard of Vertica? They do an awful lot >>of lossless compression of data in memory.
>
>No, I hadn't heard about them before. Could you point me >to some document where
>they detail how they added hardware support for compressed >memory transfers to reduce bandwidth?

They don't need hardware to do lossless compression. They have a clever column oriented database. Check vertica.com. One of their big performance gains is from reducing memory (and disk) bandwidth.

>>Many applications use adjacent values.
>
>Yes, and many applications also use non-adjacent values.
>
>If a loop contains just one load or store at an address >which isn't consecutive,
>it can't be vectorized (unless you want to resort to >serially extracting/inserting
>addresses and values). So even if the majority of values >are adjacent, it doesn't
>take a lot of non-adjacent data to cripple the performance.

You can still vectorize it, you just need to have a bunch of scalar loads/stores to deal with the non-adjacent addresses.

>>>Why? It only accesses the cache lines it needs. If all >elements are from the same
>>>cache line, it's as fast as accessing a single element.
>>
>>And exactly as fast as using AVX! i.e. no improvement >>and more complexity/power.
>
>No. The addresses are unknown at compile time. So the only >>option with AVX1 is
>to sequentially extract each address from the address >vector, and insert the read
>element into the result vector. This takes 18 instructions.

>With gather support it would be just one instruction. >Assuming it gets split into
>two 128-bit gather uops, the maximum throughput is 1 every >cycle and the minimal throughput is 1 every 4 cycles.

>>>But even in the worst case
>>>it can't generate more misses or consume more bandwidth.
>>
>>It sure can. Now instead of having 1-2 TLB accesses per cycle, you get 16. How
>>many TLB copies do you want? How many misses in flight do you want to support?
>
>You're still not getting it. It only accesses one cache >line per cycle. It simply
>has to check which elements are within the same cache >line, and perform a single
>TLB access for all of these elements. Checking whether the >addresses land on the
>same cache line doesn't require full translation of each >address.

That's quite complicated hardware, and you can't afford to have that on the critical path for any of your normal loads. So now you need a fairly separate load/store pipeline for scatter/gather.


>Nothing other than graphics runs better on the IGP. As >I've mentioned before, GPGPU
>is only succesful using high-end hardware.

Today...unclear what tomorrow holds.

>So the CPU is better than the IGP at absolutely everything >else. That makes it
>really tempting to have a closer look at what it would >take to make it adequately efficient at graphics as well.
>
>The answer: gather/scatter.

It would also need a 2X improvement in FLOP/w and /mm2, possibly more.

>Multi-core, 256-bit vectors, Hyper-Threading, software >pipelining... the CPU is
>already a throughput device! It's just being held back by >the lack of parallel load/store
>support. It's the one missing part to let all those GFLOPS >come to full fruition.

You keep on repeating this as if it were true, but it's not. I agree that lack of scatter/gather is an issue. But a more fundamental issue is that throughput optimized cores (e.g. shader arrays) are simply more efficient for compute rich workloads. You can't really get around that.

>What specialized hardware would that be? I've already shown that texture compression
>hardly makes a difference,

No, you cited extremely old data from a simulator, where even the author of the simulator thinks the data is not useful.

>and sampling and filtering is becoming programmable anyway.
>Gather/scatter speeds up just about every other pipeline >stage as well.

Except it doesn't benefit many workloads, and it costs a lot of area and power. So you want to disable it on the many workloads where it does not help.

>>Yes, but you can charge more for the system since it gets >>better battery life.
>
>No you can't, because the competition will sell it for >less and take away your market share.

What is the price delta is only $5?


>>I totally agree that scatter/gather is a great capability to have. But what's
>>the cost in die area, power and complexity? Not just to the core, but also the memory controller, etc.
>
>Larrabee has wider vectors and smaller cores, but features gather/scatter support.
>So I don't think it takes a lot of die space either way. It doesn't require any
>changes to the memory controller, just the load/store units. I'm not entirely sure
>but collecting four elements from a cache line can probably largely make use of
>the existing network to extract one (unaligned) value. And checking which addresses
>land on the same cache line is a very simple equality test >of the upper bits.

I think you have no or minimal experience designing hardware, so I'm not really inclined to take your word for it...especially compared against the expertise of the thousands of CPU designers at places like Intel, AMD and IBM.

Scatter/gather is expensive and that's why it isn't done. Even the LRB implementation was fairly limited compared to some of the older vector machines (which had both temporal and spatial scatter/gather).

David