Sizing up the Super Heavyweights

Pages: 1 2 3 4 5 6 7 8 9 10

It’s the Memory, Stupid

But higher clock frequency and SMT, are not the primary performance differentiator between the POWER4 and POWER5 architectures. Nearly ten years ago when DEC MPU designers were considering their third generation RISC design, chief Alpha architect Dick Sites reportedly summarized his analysis of the biggest short coming of the second generation EV5 chip with the concise phrase “It’s the memory, stupid”. It is clear now that IBM’s server MPU architects consider Site’s advice as relevant today as it was a decade ago. The most distinguishing characteristics of the POWER5 are the extent that IBM improved upon the POWER4+’s cache and memory system hierarchy. These changes are listed below in Table 1.

The three levels of cache in the POWER5 all have higher degrees of associativity than their counterparts in the POWER4 to reduce miss rates. In addition the on-chip L2 and external L3 are slightly larger. The biggest improvement to cache in the POWER5 is in the latency and bandwidth of the L3. Although it is still external to the processor, it now operates at 1/2 the processor clock rate in the POWER5 compared to 1/3 the processor clock rate in the POWER4. This change increases bandwidth by 50% and reduces latency by roughly a third.

The memory system in the POWER5 has been greatly improved by integrating the memory controller into the processor. In the POWER4 architecture, the memory was controlled by a separate ASIC that was accessed by the processor through the external L3 device. This change provides separate and independent paths between the MPU and L3 cache and the MPU and memory. It has the double benefit of increasing potential operational parallelism and bandwidth as well significantly reducing latency. Instead of requiring six chip-to-chip transfers and four extraneous ASIC crossings to perform a memory read, the POWER5 device directly controls up to four independent channels of DDR or DDR2, albeit through intermediary buffer chips. In a sense IBM is following the same path as Alpha EV7 and AMD Opteron architects in extracting performance gains outside of the CPU core at the system level by bringing memory closer to the processor.

The results of these optimizations are astounding – memory bandwidth per device is quadrupled even as memory latency in terms of processor clocks falls by nearly 40%. The cache and memory performance of the Madison 6M, POWER4+, and POWER5 are provided in Table 2 along with speed and throughput performance as measured by Linpack, SPEC, SPEC_rate2k, and TpmC.

The effect of lower latency and higher bandwidth is quite evident. Compared to the 1.7 GHz POWER4+ the 1.9 GHz POWER5 achieves a 30% higher SPECint_base2k score and a 61% higher SPECfp_base2k score with only a 12% higher clock frequency. The difference in throughput is even more dramatic because SPECrate2k testing on the POWER5 was run with 2N copies of the SPEC CPU suite on N processors to take advantage of its SMT capability. The unexpected magnitude of absolute improvement in cache and memory latency and bandwidth is the primary reason why the POWER5 blew past most of my performance estimates in The Battle in 64 bit Land: Merchant Chips on the Rise.

Pages: « Prev   1 2 3 4 5 6 7 8 9 10   Next »