POWER6 memory bandwidth

Article: Tukwila Update
By: Michael S (already5chosen.delete@this.yahoo.com), February 10, 2009 6:21 am
Room: Moderated Discussions
Jesper Frimann (jesperfrimann@gmail.com) on 2/9/09 wrote:
---------------------------
>Michael S (already5chosen@yahoo.com) on 2/5/09 wrote:
>---------------------------
>>As always, I'd start with Nitpicks:
>>
>>You wrote:
>>"The POWER6 in comparison has ... 75GB/s for memory"
>>
>>According to my understanding, there are two versions of POWER6 chip.
>>The first is a "normal" Power6 chip that has a standard FBD memory interface. The
>>second is a special "MCM" version that retains Power5-style elastic I/O interface
>>to IBM custom memory buffer chips. The second version is found only in p 595.
>>
>>The first version features four FBD-667 channels.
>>It means maximal read bandwidth of 8*4*.667=21.33GB/s and maximal write bandwidth
>>of 8*4*.667/2=10.67GB/s. With more than one DIMM per channel reads and writes could
>>be in theory carried simultaneously for combined figure of 32 GB/s.
>>
>>The calculations above apply to p570 populated with 8 or more DIMMs per CPU.
>>
>>Since IBM didn't publish detailed description of p560 I don't know whether the same numbers applies to it as well.
>>
>>Since IBM didn't publish detailed description of p75 I don't know whether the same
>>numbers applies to it but pretty sure that the don't. The number of supported DIMMs
>>per Power6 chip is simply not high enough to make it happen.
>>
>>p550 in theory achieves the same maximal bandwidth as p570. In practice, since
>>only 2 DIMMs could be placed on one channel achieving simultaneous read/write is less likely.
>>
>>JS12 is essentially a single-chip version of p550 so the same calculations apply.
>>
>>p520 has two memory channels per chip so max bandwidth is half of mentioned above: 10.67 GB/s read + 5.33 GB/s write.
>>
>>And finally JS22. It has two memory channels per chip but only one DIMM per channel.
>>So it can't do reads and writes simultaneously. I.e. 10.67 GB/s read or 5.33 GB/s write.
>>
>>Now let's look at the Power6-595.
>>Here each chip has 8 memory channels with one DDR2/667 DIMM per channel. In the
>>read direction data is transferred at full DIMMs speed, i.e. 8*8*.667=42.67 GB/s.
>>In the write direction there is a bottleneck in the buffer chip so the max write
>>bandwidth = 21.33 GB/s. Like in case of JS22 there is only one DIMM per channels
>>so simultaneous data transfer in both directions is not possible.
>>
>>Conclusion:
>>Even Power6-595 can't match per chip peak theoretical memory bandwidth of triple-channel DDR3-1600 Nehalem.
>>"Normal" Power6 chips are about equal to Nehalem 3x1066.
>>For entry-level Power6 gear (with exception of JS12) the peak bandwidth is lower
>>than Nehalem not only per chip but also per core.
>>One should take this facts into account when comparing SpecFp_rate results.
>
>Well I don't know if I would call it two versions. AFAIR POWER6 has 2 memory controllers each with 4 memory channels.
>And in the power 595 it uses both Memory controllers and all the channels, furthermore
>it uses 2 level 3 cache modules. And a power 595 MCM consists of 1 processor CHIP, and 2 level 3 cache modules.
>
>On the power 570 it uses all 4 memory channels but on only one memory controller.

From p570 redpaper I got the impression that CPU directly drives on-DIMM memory buffer chips.
See Figure 2-5 on p. 32.
http://www.redbooks.ibm.com/redpieces/pdfs/redp4405.pdf
I can see only three possible explanations:
1. Those are IBM custom DIMMs with IBM custom memory biffers
2. The figure is wrong. They forgot to draw protocol converter chip.
3. POWER6 used in p570 and smaller boxen has on-chip FBD memory interface, same as Niagara2 and Tukwilla. So it has to be different die from Power6 used in p595.

I picked the 3rd variant. In fact, after learning more about entry-level Power6 offerings I realize that I was almost certainly wrong. It looks like the 1st variant (custom DIMMs) is correct.


>
>And as far as I remember power 520 and js12 only uses to channels on one memory controller.

In p520 and JS22 each CPU uses half-memory controller.
JS12 is like p550 - one full controller, four channels, up to two DIMMs per channel.

http://www.redbooks.ibm.com/redbooks/pdfs/sg247655.pdf
http://www.redbooks.ibm.com/redpieces/pdfs/redp4403.pdf
http://www.redbooks.ibm.com/redpieces/pdfs/redp4404.pdf

>
>The memory subsystem section here:
>http://www.research.ibm.com/journal/rd/516/le.html
>
>(Here you can also see that the peak memory bandwidth numbers are for 800Mhz DDR2 memory).

Yes, have seen that. But, AFAIK, all publicly documented shipping systems run at 667 MHz. May be, p575 is different? It should have fewer channels than other mid-range so running faster is a good way to compensate.

>
>Now what I would really like in my work is to have a power 560 and power 570 that
>used all memory controllers and had x2 the memory. 768GB memory for the 570 is
>IMHO to little, you will run out of memory before you run out of processing power.
>
>// Jesper
>
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Tukwila Update - article onlineDavid Kanter2009/02/04 11:03 PM
  Tukwila Update - article onlineDan2009/02/05 02:17 AM
    Tukwila Update - article onlineJoe Chang2009/02/05 08:16 AM
    Tukwila Update - article onlineTemp2009/02/05 08:25 AM
      Tukwila Update - article onlinePaul2009/02/05 11:29 AM
        Tukwila Update - article onlineDavid Kanter2009/02/05 05:32 PM
          Tukwila Update - article onlinePhil2009/02/06 12:24 AM
            Great. Finally hard numbersMichael S2009/02/06 03:46 AM
            Tukwila Update - article onlinelubemark2009/02/06 04:54 AM
              Tukwila Update - article onlinePhil2009/02/06 06:29 AM
                Tukwila Update - article onlineRagingDragon2009/02/07 02:39 PM
                  Tukwila Update - article onlineMichael S2009/02/07 03:09 PM
            Tukwila Update - article onlinesavantu2009/02/06 05:23 AM
              Tukwila Update - article onlineMichael S2009/02/06 06:13 AM
            Tukwila Update - article onlinesomeone2009/02/06 06:18 AM
              Tukwila Update - article onlinePhil2009/02/06 06:47 AM
                Tukwila Update - article onlinesomeone2009/02/06 07:17 AM
                Tukwila Update - article onlineRagingDragon2009/02/07 02:51 PM
              Tukwila Update - article onlineLinus Torvalds2009/02/06 07:37 AM
                Tukwila Update - article onlinesomeone2009/02/06 08:19 AM
                Tukwila Update - article onlinesavantu2009/02/06 09:19 AM
                  Tukwila Update - article onlineLinus Torvalds2009/02/06 09:40 AM
                    Tukwila Update - article onlinesavantu2009/02/06 10:00 AM
              Tukwila Update - article onlinePhil2009/02/09 03:54 AM
                Tukwila Update - article onlineDoug Siebert2009/02/09 09:40 PM
                  Tukwila Update - article onlineJouni Osmala2009/02/10 12:03 AM
                  Tukwila Update - article onlinesomeone2009/02/10 05:15 AM
                Tukwila Update - article onlineslacker2009/02/10 05:22 PM
    Tukwila Update - article onlineMichael S2009/02/05 02:56 PM
    Tukwila Update - article onlineDavid Kanter2009/02/05 03:55 PM
    Tukwila Update - article onlinesomeone2009/02/05 04:47 PM
      Tukwila Update - article onlineanon2009/02/05 09:16 PM
        Tukwila Update - article onlineRagingDragon2009/02/05 09:27 PM
        Tukwila Update - article onlinesomeone2009/02/06 06:32 AM
          Tukwila Update - article onlineanon2009/02/06 08:25 AM
            Tukwila Update - article onlinesomeone2009/02/06 08:40 AM
  POWER6 memory bandwidthMichael S2009/02/05 02:30 AM
    POWER6 memory bandwidthsomeone2009/02/05 06:00 AM
      POWER6 memory bandwidthMichael S2009/02/05 06:36 AM
      POWER6 interconnectconfused2009/02/05 09:50 AM
        POWER6 interconnectfoobar2009/02/05 01:12 PM
    POWER6 memory bandwidthWes Felter2009/02/05 11:57 AM
    POWER6 memory bandwidthJesper Frimann2009/02/09 10:54 PM
      POWER6 memory bandwidthMichael S2009/02/10 06:21 AM
  Why the platform focus?Linus Torvalds2009/02/05 07:40 AM
    Why the platform focus?savantu2009/02/05 07:50 AM
      Why the platform focus?Vincent Diepeveen2009/02/05 08:29 AM
        Why the platform focus?savantu2009/02/05 09:34 PM
          Why the platform focus?Vincent Diepeveen2009/02/05 10:09 PM
          Why the platform focus?Phil2009/02/06 12:10 AM
            Why the platform focus?savantu2009/02/06 12:50 AM
              Why the platform focus?Phil2009/02/06 06:09 AM
                Why the platform focus?savantu2009/02/06 09:08 AM
                Why the platform focus?someone2009/02/06 09:21 AM
                  Why the platform focus?mpx2009/02/06 01:04 PM
                    Why the platform focus?someone2009/02/06 01:16 PM
                Why the platform focus?RagingDragon2009/02/07 03:16 PM
                Why the platform focus?mas2009/02/25 07:28 AM
              itanium bigger than entire car industryVincent Diepeveen2009/02/06 06:12 AM
                itanium bigger than entire car industryDevon Welles2009/02/06 06:51 AM
                  itanium bigger than entire car industryVincent Diepeveen2009/02/06 09:41 AM
                  itanium bigger than entire car industryDean Kent2009/02/06 06:56 PM
                    Unit sales is meaningless when ASP grows fastersomeone2009/02/07 08:38 AM
                      Unit sales is meaningless when ASP grows fasterDean Kent2009/02/07 02:10 PM
                        Unit sales is meaningless when ASP grows fasterRagingDragon2009/02/07 03:34 PM
                    itanium bigger than entire car industryVincent Diepeveen2009/02/08 04:35 AM
                itanium bigger than entire car industryRagingDragon2009/02/07 03:40 PM
            Why the platform focus?Vincent Diepeveen2009/02/06 06:47 AM
      Yes it doesm performance mattersbob2009/02/05 09:51 AM
        Yes it doesm performance mattersVenki2009/02/05 10:06 AM
      Why the platform focus?Doug Siebert2009/02/06 12:07 AM
        Why the platform focus?savantu2009/02/06 01:00 AM
    Why the platform focus?someone2009/02/05 09:49 AM
      Why the platform focus?Linus Torvalds2009/02/05 11:03 AM
        Why the platform focus?Default2009/02/05 12:29 PM
          Why the platform focus?anon2009/02/05 01:08 PM
        Why the platform focus?someone2009/02/05 01:24 PM
          Why the platform focus?Linus Torvalds2009/02/05 02:30 PM
    Why the platform focus?Paradox2009/02/05 10:22 AM
    Why the platform focus?slacker2009/02/05 12:41 PM
      Why the platform focus?RagingDragon2009/02/05 09:57 PM
      Why the platform focus?Michael S2009/02/06 05:11 AM
        Why the platform focus?slacker2009/02/06 12:58 PM
          Why the platform focus?Michael S2009/02/08 01:24 AM
            Why the platform focus?someone2009/02/08 08:38 AM
              Why the platform focus?David Kanter2009/02/08 03:27 PM
                Why the platform focus?someone2009/02/08 06:26 PM
                  Why the platform focus?savantu2009/02/08 11:35 PM
          Why the platform focus?someone2009/02/08 08:53 AM
    All x86 SpecInt scores are useless due to autopar (NT)Michael S2009/02/05 02:15 PM
      Auto parallelizationDavid Kanter2009/02/05 05:17 PM
      All x86 SpecInt scores are useless due to autopar (NT)Paradox2009/02/06 07:47 AM
    Why the platform focus?David Kanter2009/02/05 03:49 PM
    Why the platform focus?David Kanter2009/02/06 12:09 AM
      Why the platform focus?Linus Torvalds2009/02/06 07:14 AM
        Why the platform focus?savantu2009/02/06 09:37 AM
          Why the platform focus?Linus Torvalds2009/02/06 11:49 AM
            Why the platform focus?Linus Torvalds2009/02/06 12:09 PM
            Intel puts its money where its mouth issomeone2009/02/06 01:08 PM
              Intel puts its money where its mouth isRagingDragon2009/02/07 05:01 PM
                Intel puts its money where its mouth issomeone2009/02/08 01:24 PM
                  mission-criticalMichael S2009/02/08 04:06 PM
                    mission-criticalmpx2009/02/09 01:30 AM
                      mission-criticalrwessel2009/02/09 02:23 PM
                    mission-criticalanon2009/02/09 02:55 AM
                    mission-criticalEduardoS2009/02/09 04:17 PM
                      mission-criticalDean Kent2009/02/09 07:11 PM
                        mission-criticalMichael S2009/02/10 04:20 AM
                          mission-criticalDean Kent2009/02/10 06:26 AM
                            mission-criticalMichael S2009/02/10 07:01 AM
                              mission-criticalDean Kent2009/02/10 12:36 PM
                            mission-criticalsomeone2009/02/10 08:05 AM
                              mission-criticalDean Kent2009/02/10 12:22 PM
                                mission-criticalZt2009/02/22 03:54 PM
                          mission-criticalanon2009/02/10 09:41 PM
                        mission-criticalEduardoS2009/02/10 12:46 PM
                          mission-criticalDean Kent2009/02/10 01:31 PM
                        mission-criticalslacker2009/02/10 06:30 PM
                    mission-criticalVincent Diepeveen2009/02/18 06:20 AM
                  Mission criticalmpx2009/02/09 12:00 AM
            Why the platform focus?savantu2009/02/07 12:15 AM
              Sun and x86 server differentiationDavid Kanter2009/02/07 12:34 AM
                Sun and x86 server differentiationmax2009/02/07 02:30 AM
                Sun and x86 server differentiationsomeone2009/02/07 09:19 AM
                  Sun and x86 server differentiationLinus Torvalds2009/02/07 09:44 AM
                Sun and x86 server differentiationRagingDragon2009/02/07 05:09 PM
                  Sun and x86 server differentiationMichael S2009/02/08 04:05 AM
                    Sun and x86 server differentiationRagingDragon2009/02/09 11:03 PM
                    Sun and x86 server differentiationJesper Frimann2009/02/09 11:51 PM
                      Sun and x86 server differentiationAlex Jones2009/02/10 12:43 PM
            Why the platform focus?bob2009/02/08 03:51 AM
              Why the platform focus?someone2009/02/08 08:23 AM
          missing the big pictureAM2009/02/18 05:43 AM
            missing the big pictureMichael S2009/02/18 07:42 AM
              missing the big pictureAM2009/02/18 08:03 AM
        Why the platform focus?mpx2009/02/06 11:47 AM
          Itanium - slowest and most obsolete server CPU family in the world, NOW.mpx2009/02/06 03:48 PM
            Itanium - slowest and most obsolete server CPU family in the world, NOW.Paul2009/02/07 01:56 PM
              z series?Michael S2009/02/07 02:12 PM
              Itanium - slowest and most obsolete server CPU family in the world, NOW.someone else2009/02/24 03:37 AM
                Itanium - slowest and most obsolete server CPU family in the world, NOW.EduardoS2009/02/24 05:55 AM
                  Itanium - slowest and most obsolete server CPU family in the world, NOW.someone else2009/02/25 12:55 AM
                    Itanium - slowest and most obsolete server CPU family in the world, NOW.Michael S2009/02/25 01:27 AM
          Why the platform focus?RagingDragon2009/02/07 05:18 PM
            Why the platform focus?Paul2009/02/08 12:10 PM
              Why the platform focus?Jukka Larja2009/02/08 10:04 PM
        Why the platform focus?slacker2009/02/06 01:10 PM
          Why the platform focus?Linus Torvalds2009/02/06 01:40 PM
            Why the platform focus?savantu2009/02/06 01:51 PM
            Why the platform focus?someone2009/02/06 01:58 PM
              Why the platform focus?Linus Torvalds2009/02/07 08:26 AM
                Why the platform focus?someone2009/02/07 09:10 AM
                  Why the platform focus?Linus Torvalds2009/02/07 09:40 AM
                    Why the platform focus?someone2009/02/07 11:24 AM
                      Why the platform focus?Doug Siebert2009/02/07 11:32 PM
                        Why the platform focus?max2009/02/08 03:57 AM
                          Why the platform focus?Michael S2009/02/08 04:20 AM
                        Why the platform focus?someone2009/02/08 08:15 AM
                          Why the platform focus?Doug Siebert2009/02/08 10:36 PM
                      Why the platform focus?hobold2009/02/09 04:49 AM
                Why the platform focus?anon2009/02/24 12:57 AM
                  Why the platform focus?Linus Torvalds2009/02/24 08:45 AM
                    Why the platform focus?savantu2009/02/24 11:30 AM
                      Why the platform focus?slacker2009/02/24 12:51 PM
                        Why the platform focus?savantu2009/02/24 11:04 PM
                          Why the platform focus?Michael S2009/02/25 01:34 AM
                          Why the platform focus?anon2009/02/25 09:17 AM
                      Why the platform focus?max2009/02/25 10:15 AM
                    Why the platform focus?someone2009/02/24 04:43 PM
                      Why the platform focus?Doug Siebert2009/02/24 07:26 PM
                        Why the platform focus?Howard Chu2009/02/25 02:07 AM
                          Why the platform focus?someone2009/02/25 05:48 AM
                        Why the platform focus?someone2009/02/25 05:41 AM
                          Why the platform focus?Linus Torvalds2009/02/25 08:17 AM
                            Why the platform focus?someone2009/02/25 08:55 AM
                              has anyone seen Tukwila silicon?anon2009/02/25 09:38 AM
                              Why the platform focus?Linus Torvalds2009/02/25 10:05 AM
                                Why the platform focus?slacker2009/02/25 12:11 PM
                                Why the platform focus?a reader2009/02/26 08:11 PM
                                  Why the platform focus?rcf2009/02/27 12:32 PM
                                    Why the platform focus?max2009/02/27 01:11 PM
                                      Why the platform focus?rcf2009/02/27 02:50 PM
                            Why the platform focus?Vincent Diepeveen2009/02/25 03:30 PM
                            $40M sale to $16M companybob2009/02/25 07:25 PM
                              $40M sale to $16M companyRichard Cownie2009/02/26 11:21 AM
                  Why the platform focus?anonymous2009/02/24 10:52 AM
                    Why the platform focus?savantu2009/02/24 11:20 AM
                      Why the platform focus?anonymous2009/02/24 02:31 PM
                        Why the platform focus?savantu2009/02/24 11:05 PM
                        Why the platform focus?someone else2009/02/25 12:04 AM
                          Why the platform focus?Michael S2009/02/25 12:42 AM
                            Put me down for $500 that Poulson doesn't arrive earlier than Q4/2011 (NT)slacker2009/02/25 11:39 AM
                        Why the platform focus?someone2009/02/25 05:54 AM
                          Why the platform focus?anonymous2009/02/25 08:46 AM
                            Why the platform focus?someone2009/02/25 09:22 AM
                            Why the platform focus?anon2009/02/25 10:01 AM
                              Why the platform focus?anonymous2009/02/25 10:54 AM
                  Why the platform focus?mpx2009/02/24 01:11 PM
                    Why the platform focus?anon2009/02/24 07:57 PM
                      Why the platform focus?Doug Siebert2009/02/24 09:04 PM
                        Why the platform focus?anon2009/02/24 09:46 PM
                          Why the platform focus?Doug Siebert2009/02/25 04:13 PM
                            Why the platform focus?anon2009/02/25 07:53 PM
                              Why the platform focus?bob2009/02/25 08:00 PM
                                Please try to keep up (NT)anon2009/02/25 08:49 PM
                              Why the platform focus?Doug Siebert2009/02/25 11:09 PM
                                Why the platform focus?anon2009/02/26 12:12 AM
                                Why the platform focus?Michael S2009/02/26 01:16 AM
                                  Why the platform focus?James2009/02/26 05:09 AM
                                    sufficiently intimate with the OSMichael S2009/02/26 05:29 AM
                                      sufficiently intimate with the OSanon2009/02/27 12:01 AM
                                      sufficiently intimate with the OSHoward Chu2009/02/27 12:37 AM
                      Why the platform focus?Michael S2009/02/25 01:02 AM
                        Why the platform focus?anon2009/02/25 02:07 AM
              Why the platform focus?anon2009/02/07 12:18 PM
                Why the platform focus?Vincent Diepeveen2009/02/08 09:16 AM
                Why the platform focus?anon2009/02/25 06:40 AM
                  Intels financial statusVincent Diepeveen2009/02/25 11:02 AM
                Why the platform focus?someone2009/02/25 06:54 AM
      Why the platform focus?Vincent Diepeveen2009/02/06 07:20 AM
        Why the platform focus?Default2009/02/06 08:57 AM
          Why the platform focus?Vincent Diepeveen2009/02/06 09:59 AM
          Why the platform focus?RagingDragon2009/02/07 05:43 PM
  Tukwila Update - article onlineVincent Diepeveen2009/02/05 08:11 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell avocado?