Emil Briggs (me.delete@this.nowherespam.com) on July 28, 2012 6:11 pm wrote:

> There are two places where we are
> using GPU's. One of them consists of large matrix operations and are done using
> the Nvidia cublas library. Those run close to 80% of peak. The tricky part of
> the work here is keeping the CPU's busy doing something useful while moving the
> matrices back and forth to the GPU. The other place is some finite difference
> routines. Still working on this. It's faster than doing it all on the CPU's but
> not by much. I'm trying to get more overlap between the CPU and GPU's here but
> this section of the code is not as suitable for that as the first.
>

80% is very very good. Esp compared to linpack.

>
> Agreed. How difficult do you think it would be to implement
> coherent memory access?
>

Depends. If you can convince PCI SIG to implement a coherent protocol the difficulty shouldn't be that high. Efficiency wouldn't be the greatest since the basic protocols for PCI-E are designed around large block transfers but it would be doable esp with the move to integrating PCI-E on die.

Probably the easiest solution for something like MIC would be to integrate a QPI interface in addition to the PCI-E interface. At that point it primarily becomes an exercise in setting up the memory map reasonably/sanely. The MIC would need both a caching agent and a home coherency agent, but it should be possible to do some cut and paste.

What would be more difficult in the QPI + PCI-E space is get maximum advantage out of it. You would ideally like to use both the QPI agent and the PCI-E agent for bulk DMA traffic while only using the QPI agent for coherent traffic. Using the QPI link for bulk DMA would likely take some work with the various DMA engines. For the MIC local coherent memory you would likely only make a subset of it available for coherent access from the CPU in order to simplify the performance requirements (likely a variable window size that is programmable) so that all memory accesses from the MIC to its local memory don't have to remote snoop though if you have the area available, you might be able to get away with an SRAM based directory (basically limited capacity coherency from the MIC to CPU aka evict to make space) as well.

For the CPU memory, you would likely be unrestricted assuming that the CPU side used some form of directory.

Total DMA bandwidth should be at least equal to 32x PCI-E. And for coherent access you are looking at a minimum of 16x PCI-E bandwidth.

And from a practical standpoint you are going to want 2xQPI or QPI+PCI-E since it is unlikely that the market requirement will be there for the CPUs to have 3x 16x PCI-E. Though if your network interface chip runs over a single QPI link it might be viable, but I kinda see the ideal setup for a top end super as 1 QPI + 16x PCI-E to both the MIC/GPU and to the network interface. So you would be looking at ~32+GB/s (at current speeds, likely 64+ GB/s in the 2015 timeframe baed on PCI-E 4.0 announced goals) in and out of the CPU to both network and MIC for a total of 128 GB/s which means that memory bandwidth likely becomes you main bottleneck.

Also lets not forget that by the time this happens we are likely going to see some form of stacked memory in reasonably wide use, which means that the CPUs will likely have 1-4 GB of ultra high bandwidth "cache". Which if used right would provide enough bandwidth buffer to have the I/Os plus having the option to direct route to/from networkMIC would make the 102.4 GB/s CPU memory subsystem reasonable.

The next big problem is going to be feeding the network bandwidth. With that type of IO capability you would need 9+ 4x FDR IB connections per node. And you're probably going to want a switchless topology, can that would be a lot of switches.