Gabriele Svelto (gabriele.svelto@gmail.com) on 1/5/11 wrote:
---------------------------
>Nicolas Capens (nicolas.capens@gmail.com) on 1/5/11 wrote:
>---------------------------
>>Gabriele Svelto (gabriele.svelto@gmail.com) on 1/5/11 wrote:
>>I really don't think you can blame Knights Ferry's thermal envelop on gather/scatter
>>support. For starters, on a 45 nm process it's a massive chip by any standard. And
>>with 32 cores and 512-bit vectors, there's an intense switching activity throughout the entire chip.
>
>I suppose scatter-gather support is not the only problem, but it's there and it's pretty complex.
>

Do you happen to have any sources which explain just how complex it is? I'm sure it's not trivial, but given that LRBni already has many other features which I'd consider relatively complex, I'd be surprised if gather/scatter took a disproportionately large area. Intel was able to fit 32 of these feature rich cores with 512-bit vectors onto a chip roughly double the transistor count of Sandy Bridge...

>>Besides, Larrabee needs one large 16x32-bit crossbar, while an optimal gather/scatter
>>implementation for Sandy Bridge would only require two 4x32-bit crossbars (plus support for unaligned accesses).
>
>To do a full gather AVX load with arbitrary addresses in once cycle (assuming all
>the addresses are in the same cache-line) you would need much more than two 4x32-bits
>crossbars. You'd need 8x32-bits and you have to cover situations where all the source
>addresses the same entry which requires broadcasting over destination elements,
>not pretty in hardware, those things scale quadratically.
>

What I meant is that the two 128-bit load units could each have their own 4x32-bit crossbar. It means they sometimes load the same cache line, but that's ok. I assume that's what already happens anyway with vmovaps, and it considerably simplifies the crossbar. So while Larrabee requires one massive 16x32 crossbar an architecture based on Sandy Bridge would have a near optimal gather/scatter implementation with two simpler 4x32 crossbars.

>>Since ordinary loads and stores already use a crossbar for selecting 8, 16, 32,
>>64 or 128-bit from any position in the cache line, I have a hard time believing
>>that adding the ability to select four 32-bit elements would be all that much more complicated.
>
>AFAIK they do not use a cross-bar, they don't have too, you are dealing with only
>two values and some shifting and masking.
>

Ah that's right. I hadn't completely thought that through. It probably means that the 4x32-bit crossbars also wouldn't be much more complicated to support unaligned accesses.

>>And while I agree the L1 latency should be kept minimal, Sandy Bridge's massive
>>overclocking potential leads me to believe the timings aren't that critical. In
>>the future smaller processes should have even more headroom as the clock frequency
>>is held back by thermal limitations, not timings.
>
>Yes but having a large crossbar there makes a significant difference. That's exactly
>the reason why - in spite of it's lower clock - Knight's Ferry doesn't support full
>permutations over its vectors. The resulting hardware would be enormous.
>

Yes, there's not a lot of use to support every single static permutation anyway. There's only a handful of them that are worth the hardware, and even if you do need a more complicated permutation you can simply combine them.

Note that with two 128-bit load units you can have equally small crossbars as for supporting pshufd, to efficiently implement gather/scatter.

>>If it does turn out to be an issue, maybe it can have a fast path for ordinary
>>loads/stores, and a higher latency path for gather/scatter. Again, it doesn't have
>>to be totally optimal to already start breaking down Amdahl's wall.
>
>That would make a lot more sense.
>
>>In the long term the GPU will inevitably be unified with the CPU cores. Sandy Bridge's
>>CPU already has more computing power than its GPU. And efficient software renderers
>>such as SwiftShader can outperform a range of IGPs and low-end GPUs. The only thing
>>holding back the CPU from becoming really efficient at graphics is gather/scatter support.
>
>At the same image-quality level? I'd like to see a pointer to that.

At the same or higher image quality level. On a Core i7 965 SwiftShader scores 620 3DMark06 points. That's more than a GMA X3100.

> Besides as
>Larrabee's demise has proven that bolting large vector units and adding scatter/gather
>to CPU cores is not enough for graphics from a performance standpoint and an even
>poorer choice from a performance/W one. Specialized hardware in GPUs is here to
>stay and adapting that to be used in a CPU-centric environment is not going to be an easy task.
>

CPUs have always run tasks which they're not specifically designed for. There's simply a point where a finely tuned generic processor is more interesting than a highly specialized one with limitations. As I've mentioned before, GPUs have also unified their vertex and pixel pipelines into more generic shader cores, because with the latter you can run a much larger range of workloads efficiently. It also spurs creativity to be able to go beyond what everyone else has been using the hardware for.

Also note that there's like a 100 fold difference in performance between the high-end and low-end GPUs on the market today. So clearly the consumer's expectations vary wildly. People who don't really care about gaming (except maybe causal games), would be better off with extra CPU cores for higher performance in the applications they do care about. Efficient software renderers running on modern CPUs can already take care of occasional 3D applications (3D on the web will be a huge thing - Adobe has licensed SwiftShader for future versions of Flash, for when the hardware is inadequate). And despite the fact that cutting edge games have an insational hunger for faster GPUs, everything else is not a fast moving target. Today's casual games aren't all that much heavier than those from several years ago, while in the same time period mainstream CPUs have become many times faster.

At the same time, GPUs also continue to become more generic. The days of fixed-function rendering are long gone and games continously push the boundaries of the hardware. NVIDIA has managed to create an architecture which can outperform an AMD GPU, with only half the theoretical GFLOPS. So clearly other factors start to matter a lot. GPUs need ever larger caches, reduced branch granularities, ALUs with more generic operations, support for call stacks, etc. Mark my words, one day they'll need speculative execution just to prevent the register file from growing out of proportion. Likely before that we'll also have programmable texture filtering, which means the texture units become more like generic gather/scatter units and the actual filtering is done at full FP32 precision in the shader cores (AMD already does the texel address calculations in the shader). It's all evolving toward more generic capabilities, because the workloads people expect their GPU to execute are ever more diverse. This is bound to collide with the CPUs space one day. It's no coincidence that NVIDIA has desperately attempted to acquire an x86 licence. It's low-end market is threatened by IGPs, and IGPs themselves are threatened by software renderers (which isn't an issue for CPU manufacturers but makes NVIDIA panic over losing grip on the GPGPU market and eventually its GPU market).

The only reason we haven't seen a sign of Larrabee as a GPU yet, is because it's too soon, and because it approaches the market from the wrong end. It's too soon because despite the enormous opportunities a generic architecture like that offers, developers aren't just going to invest in developing applications for it if it has too little market share. And it has too little market share because Intel tried to sell it as a high-end GPU. It's an architecture with huge potential but it's facing a massive chicken-and-egg problem. We first need the rest of the GPU landscape to evolve in the same direction before developers will take the plunge. There's simply no strong software ecosystem yet for things that go beyond the classic graphics APIs. OpenCL and such are cautious first steps but it will take many more years before you can write code in the language of your liking and have it run on either a CPU or a GPU. It takes more steps toward full convergence for this to become a reality.

Software rendering for the low-end market on the other hand is viable today. There's bound to be people who rather have a more powerful CPU instead of an on-die GPU. And these powerful CPUs are attractive to developers because they can safely extend the existing software ecosystem without big risky investments. This software further increases the demand for powerful CPUs, and these will be capable of more than just low-end graphics...

If all this still seems unlikely to you, just look at what happened to sound processing. In the early days, mixing and modulating multiple channels at high quality was a heavy task for the CPU to perform. So your best option was to get a discrete sound card. Then this sound processor migrated to the chipset. And nowadays it has been reduced to just an I/O chip and the actual processing takes place in an audio codec run by the CPU, thanks to its vastly increased performance. Not even power efficiency can reverse that trend.

The same thing will happen with graphics. Sandy Bridge already makes the GPU share the CPU's caches and memory controllers. So it's not that big a step to perform the actual processing on the CPU cores. Once the CPU can run Crysis at high quality, which I expect to happen well within this decade, the number of people willing to pay for a GPU will have reduced considerably. And this entices more developers to make use of the powerful generic capabilities the CPU offers, meaning that this software doensn't run very efficiently on a GPU unless it evolves even closer to the CPU's architecture. It's an unstoppable spiral which will cause the GPU to go the way of the dodo in the long end, just like sound cards are exhaling their last breath.

>>Currently either the CPU or the GPU is a bottleneck. By ditching the GPU and adding
>>more generic CPU cores instead, with gather/scatter support, this bottleneck would
>>be gone (just like the vertex/pixel bottleneck dissapeared with unified GPU architectures).
>
>That's not so easy. Coherent memory will always impose a burden on you and it just
>gets worse if you're MLP is very high like in graphics workloads. And besides that
>you still want dedicated hardware for filtering, rasterizing and friends. Dealing
>with that in a sane way in a general purpose environment is no easy task.
>

The DLP for graphics workloads is stagnating. The motto "batch, batch, batch" has reached its limits and to efficiently run more diverse applications you need to take advantage of more than just data level parallelism. The CPU takes advantage of ILP, TLP and DLP. GPUs have only very recently started to take advantage of ILP and TLP. This will become more important in the future, and it costs compute density. The CPU on the other hand is still increasing DLP using wider vectors, FMA and core modules (AMD Bulldozer).

And like I said before we don't want dedicated hardware for filtering. Everything is pointing towards more generic gather/scatter units and filtering in the shaders. Neither do we want dedicated hardware for rasterization. Tesselation is an early step toward programmable rasterization.

None of this is happening overnight and indeed it's not an easy task, but if you look what GPUs were like several years ago, and extrapolate that several years into the future, it's readily apparent that not a single graphics-specific feature will survive the test of time.

>>Note that giving the GPU a sane programming model requires massive changes which
>>bring it much closer to a CPU architecture and lower computing density.
>
>Not really, making GPUs start to look like long-vector co-processors is probably
>going to be achievable in a fairly short time frame having them on the die, and
>that would be a pretty sane programming model. The memory model remains the largest
>problem IMHO, even with a shared memory pool.

Please name one co-processor on the same die, which after many years still remains a heterogeneous component.

The fact of the matter is that no developer really likes heterogenous architectures. Software development is complex enough as it is to avoid having to deal with multiple programming models. Also keep in mind that the GPU's programming models haven't even settled down. Supporting multiple brands and even multiple generations from the same manufacturer is a huge pain. So together with the 100 fold difference in performance between the high-end and low-end, we're nowhere near using the GPU as a reliable vector co-processor. Targetting SSE and AVX offers a lot more guarantees.

The only thing missing to make the CPU's SIMD support complete, is gather/scatter. Once that's available, the GPU makes little chance as a co-processor.