By: anon (anon.delete@this.anon.com), July 29, 2012 11:09 am
Room: Moderated Discussions
aaron spink (aaronspink.delete@this.notearthlink.net) on July 29, 2012 5:39 am wrote:
> Emil Briggs (me.delete@this.nowherespam.com) on July 28, 2012 6:11 pm
> wrote:
>
> > There are two places where we are
> > using GPU's. One of them
> consists of large matrix operations and are done using
> > the Nvidia cublas
> library. Those run close to 80% of peak. The tricky part of
> > the work here
> is keeping the CPU's busy doing something useful while moving the
> >
> matrices back and forth to the GPU. The other place is some finite difference
>
> > routines. Still working on this. It's faster than doing it all on the
> CPU's but
> > not by much. I'm trying to get more overlap between the CPU and
> GPU's here but
> > this section of the code is not as suitable for that as
> the first.
> >
>
> 80% is very very good. Esp compared to linpack.
>
> >
>
> > Agreed. How difficult do you think it would be to implement
> >
> coherent memory access?
> >
>
> Depends. If you can convince PCI SIG to
> implement a coherent protocol the difficulty shouldn't be that high. Efficiency
> wouldn't be the greatest since the basic protocols for PCI-E are designed around
> large block transfers but it would be doable esp with the move to integrating
> PCI-E on die.
>
> Probably the easiest solution for something like MIC would be
> to integrate a QPI interface in addition to the PCI-E interface. At that point
> it primarily becomes an exercise in setting up the memory map reasonably/sanely.
> The MIC would need both a caching agent and a home coherency agent, but it
> should be possible to do some cut and paste.
>
> What would be more difficult in
> the QPI + PCI-E space is get maximum advantage out of it. You would ideally
> like to use both the QPI agent and the PCI-E agent for bulk DMA traffic while
> only using the QPI agent for coherent traffic. Using the QPI link for bulk DMA
> would likely take some work with the various DMA engines. For the MIC local
> coherent memory you would likely only make a subset of it available for coherent
> access from the CPU in order to simplify the performance requirements (likely a
> variable window size that is programmable) so that all memory accesses from the
> MIC to its local memory don't have to remote snoop though if you have the area
> available, you might be able to get away with an SRAM based directory (basically
> limited capacity coherency from the MIC to CPU aka evict to make space) as well.
>
>
> For the CPU memory, you would likely be unrestricted assuming that the CPU
> side used some form of directory.
>
> Total DMA bandwidth should be at least
> equal to 32x PCI-E. And for coherent access you are looking at a minimum of 16x
> PCI-E bandwidth.
>
> And from a practical standpoint you are going to want 2xQPI
> or QPI+PCI-E since it is unlikely that the market requirement will be there for
> the CPUs to have 3x 16x PCI-E. Though if your network interface chip runs over
> a single QPI link it might be viable, but I kinda see the ideal setup for a top
> end super as 1 QPI + 16x PCI-E to both the MIC/GPU and to the network interface.
> So you would be looking at ~32+GB/s (at current speeds, likely 64+ GB/s in the
> 2015 timeframe baed on PCI-E 4.0 announced goals) in and out of the CPU to both
> network and MIC for a total of 128 GB/s which means that memory bandwidth likely
> becomes you main bottleneck.
>
> Also lets not forget that by the time this
> happens we are likely going to see some form of stacked memory in reasonably
> wide use, which means that the CPUs will likely have 1-4 GB of ultra high
> bandwidth "cache".
It does not have to be a cache of main memory; the stacked DRAM could be (part of) main memory itself. Why not?
> Which if used right would provide enough bandwidth buffer to
> have the I/Os plus having the option to direct route to/from networkMIC would
> make the 102.4 GB/s CPU memory subsystem reasonable.
>
> The next big problem is
> going to be feeding the network bandwidth. With that type of IO capability you
> would need 9+ 4x FDR IB connections per node. And you're probably going to want
> a switchless topology, can that would be a lot of switches.
> Emil Briggs (me.delete@this.nowherespam.com) on July 28, 2012 6:11 pm
> wrote:
>
> > There are two places where we are
> > using GPU's. One of them
> consists of large matrix operations and are done using
> > the Nvidia cublas
> library. Those run close to 80% of peak. The tricky part of
> > the work here
> is keeping the CPU's busy doing something useful while moving the
> >
> matrices back and forth to the GPU. The other place is some finite difference
>
> > routines. Still working on this. It's faster than doing it all on the
> CPU's but
> > not by much. I'm trying to get more overlap between the CPU and
> GPU's here but
> > this section of the code is not as suitable for that as
> the first.
> >
>
> 80% is very very good. Esp compared to linpack.
>
> >
>
> > Agreed. How difficult do you think it would be to implement
> >
> coherent memory access?
> >
>
> Depends. If you can convince PCI SIG to
> implement a coherent protocol the difficulty shouldn't be that high. Efficiency
> wouldn't be the greatest since the basic protocols for PCI-E are designed around
> large block transfers but it would be doable esp with the move to integrating
> PCI-E on die.
>
> Probably the easiest solution for something like MIC would be
> to integrate a QPI interface in addition to the PCI-E interface. At that point
> it primarily becomes an exercise in setting up the memory map reasonably/sanely.
> The MIC would need both a caching agent and a home coherency agent, but it
> should be possible to do some cut and paste.
>
> What would be more difficult in
> the QPI + PCI-E space is get maximum advantage out of it. You would ideally
> like to use both the QPI agent and the PCI-E agent for bulk DMA traffic while
> only using the QPI agent for coherent traffic. Using the QPI link for bulk DMA
> would likely take some work with the various DMA engines. For the MIC local
> coherent memory you would likely only make a subset of it available for coherent
> access from the CPU in order to simplify the performance requirements (likely a
> variable window size that is programmable) so that all memory accesses from the
> MIC to its local memory don't have to remote snoop though if you have the area
> available, you might be able to get away with an SRAM based directory (basically
> limited capacity coherency from the MIC to CPU aka evict to make space) as well.
>
>
> For the CPU memory, you would likely be unrestricted assuming that the CPU
> side used some form of directory.
>
> Total DMA bandwidth should be at least
> equal to 32x PCI-E. And for coherent access you are looking at a minimum of 16x
> PCI-E bandwidth.
>
> And from a practical standpoint you are going to want 2xQPI
> or QPI+PCI-E since it is unlikely that the market requirement will be there for
> the CPUs to have 3x 16x PCI-E. Though if your network interface chip runs over
> a single QPI link it might be viable, but I kinda see the ideal setup for a top
> end super as 1 QPI + 16x PCI-E to both the MIC/GPU and to the network interface.
> So you would be looking at ~32+GB/s (at current speeds, likely 64+ GB/s in the
> 2015 timeframe baed on PCI-E 4.0 announced goals) in and out of the CPU to both
> network and MIC for a total of 128 GB/s which means that memory bandwidth likely
> becomes you main bottleneck.
>
> Also lets not forget that by the time this
> happens we are likely going to see some form of stacked memory in reasonably
> wide use, which means that the CPUs will likely have 1-4 GB of ultra high
> bandwidth "cache".
It does not have to be a cache of main memory; the stacked DRAM could be (part of) main memory itself. Why not?
> Which if used right would provide enough bandwidth buffer to
> have the I/Os plus having the option to direct route to/from networkMIC would
> make the 102.4 GB/s CPU memory subsystem reasonable.
>
> The next big problem is
> going to be feeding the network bandwidth. With that type of IO capability you
> would need 9+ 4x FDR IB connections per node. And you're probably going to want
> a switchless topology, can that would be a lot of switches.
