Jesper Frimann (jesperfrimann@gmail.com) on 2/9/09 wrote:
---------------------------
>Michael S (already5chosen@yahoo.com) on 2/5/09 wrote:
>---------------------------
>>As always, I'd start with Nitpicks:
>>
>>You wrote:
>>"The POWER6 in comparison has ... 75GB/s for memory"
>>
>>According to my understanding, there are two versions of POWER6 chip.
>>The first is a "normal" Power6 chip that has a standard FBD memory interface. The
>>second is a special "MCM" version that retains Power5-style elastic I/O interface
>>to IBM custom memory buffer chips. The second version is found only in p 595.
>>
>>The first version features four FBD-667 channels.
>>It means maximal read bandwidth of 8*4*.667=21.33GB/s and maximal write bandwidth
>>of 8*4*.667/2=10.67GB/s. With more than one DIMM per channel reads and writes could
>>be in theory carried simultaneously for combined figure of 32 GB/s.
>>
>>The calculations above apply to p570 populated with 8 or more DIMMs per CPU.
>>
>>Since IBM didn't publish detailed description of p560 I don't know whether the same numbers applies to it as well.
>>
>>Since IBM didn't publish detailed description of p75 I don't know whether the same
>>numbers applies to it but pretty sure that the don't. The number of supported DIMMs
>>per Power6 chip is simply not high enough to make it happen.
>>
>>p550 in theory achieves the same maximal bandwidth as p570. In practice, since
>>only 2 DIMMs could be placed on one channel achieving simultaneous read/write is less likely.
>>
>>JS12 is essentially a single-chip version of p550 so the same calculations apply.
>>
>>p520 has two memory channels per chip so max bandwidth is half of mentioned above: 10.67 GB/s read + 5.33 GB/s write.
>>
>>And finally JS22. It has two memory channels per chip but only one DIMM per channel.
>>So it can't do reads and writes simultaneously. I.e. 10.67 GB/s read or 5.33 GB/s write.
>>
>>Now let's look at the Power6-595.
>>Here each chip has 8 memory channels with one DDR2/667 DIMM per channel. In the
>>read direction data is transferred at full DIMMs speed, i.e. 8*8*.667=42.67 GB/s.
>>In the write direction there is a bottleneck in the buffer chip so the max write
>>bandwidth = 21.33 GB/s. Like in case of JS22 there is only one DIMM per channels
>>so simultaneous data transfer in both directions is not possible.
>>
>>Conclusion:
>>Even Power6-595 can't match per chip peak theoretical memory bandwidth of triple-channel DDR3-1600 Nehalem.
>>"Normal" Power6 chips are about equal to Nehalem 3x1066.
>>For entry-level Power6 gear (with exception of JS12) the peak bandwidth is lower
>>than Nehalem not only per chip but also per core.
>>One should take this facts into account when comparing SpecFp_rate results.
>
>Well I don't know if I would call it two versions. AFAIR POWER6 has 2 memory controllers each with 4 memory channels.
>And in the power 595 it uses both Memory controllers and all the channels, furthermore
>it uses 2 level 3 cache modules. And a power 595 MCM consists of 1 processor CHIP, and 2 level 3 cache modules.
>
>On the power 570 it uses all 4 memory channels but on only one memory controller.

From p570 redpaper I got the impression that CPU directly drives on-DIMM memory buffer chips.
See Figure 2-5 on p. 32.
http://www.redbooks.ibm.com/redpieces/pdfs/redp4405.pdf
I can see only three possible explanations:
1. Those are IBM custom DIMMs with IBM custom memory biffers
2. The figure is wrong. They forgot to draw protocol converter chip.
3. POWER6 used in p570 and smaller boxen has on-chip FBD memory interface, same as Niagara2 and Tukwilla. So it has to be different die from Power6 used in p595.

I picked the 3rd variant. In fact, after learning more about entry-level Power6 offerings I realize that I was almost certainly wrong. It looks like the 1st variant (custom DIMMs) is correct.


>
>And as far as I remember power 520 and js12 only uses to channels on one memory controller.

In p520 and JS22 each CPU uses half-memory controller.
JS12 is like p550 - one full controller, four channels, up to two DIMMs per channel.

http://www.redbooks.ibm.com/redbooks/pdfs/sg247655.pdf
http://www.redbooks.ibm.com/redpieces/pdfs/redp4403.pdf
http://www.redbooks.ibm.com/redpieces/pdfs/redp4404.pdf

>
>The memory subsystem section here:
>http://www.research.ibm.com/journal/rd/516/le.html
>
>(Here you can also see that the peak memory bandwidth numbers are for 800Mhz DDR2 memory).

Yes, have seen that. But, AFAIK, all publicly documented shipping systems run at 667 MHz. May be, p575 is different? It should have fewer channels than other mid-range so running faster is a good way to compensate.

>
>Now what I would really like in my work is to have a power 560 and power 570 that
>used all memory controllers and had x2 the memory. 768GB memory for the 570 is
>IMHO to little, you will run out of memory before you run out of processing power.
>
>// Jesper
>