So What is the Issue Exactly?
Figure 1 – XDR memory system with 16 bit wide data bus
Figure 1 above illustrates an XDR memory system with a 16 bit wide data bus (For the purpose of illustration only. The XDR memory system on the CELL processor has two physical channels that are 36 bits wide per channel). Rambus’s XDR memory system is an interesting memory system in that it heavily leverages the Flexphase de-skewing mechanism to ensure correctness of operation. The Flexphase de-skewing mechanism relies on chains of per-bit-lane serial shift registers that can buffer the data bus input of the XDR memory system on a bit-by-bit basis. With data buffered at the interface of the memory controller, the Flexphase de-skewing circuitry can then (within reason) remove data skew introduced by differences in memory system topology and signal path lengths, and provide synchronized parallel data to the internal databus of the XDR memory controller. Since the CELL processor implements the XDR memory system, it must by default include the Flexphase circuitry to filter out some of the expected skew.
However, in order to operate with more DRAM devices in a given channel of XDR memory systems, the Flexphase circuitry in the controller interface of the XDR memory system must be able to filter out the additional skew.
Figure 2 – XDR Memory System Exhibiting Intra-Device and Inter-Device Skew on Data Bus
Figure 2 shows that in the XDR memory system, command and address information is delivered to the DRAM devices across the 12 bit wide command and address bus that operates at the data rate of 800 Mbps. Figure 2 also shows that due to the physical speed of signal propagation, DRAM device 0 receives the command and address ahead of device 1. As a result, device 0 places the data from the requested location onto the data bus at an earlier time than device 1, thus introducing skew between different bits on the data bus. This skew is referred to as inter-device skew. Figure 2 further illustrates that due to electrical and path length differences, bits from the same device will exhibit some amount of skew. This skew is referred to as intra-device skew. The XDR memory system expects that with the memory system operating at multi-Gbit data rates, inter-device as well as intra-device data skew will be the norm rather than the exception; Flexphase is designed specifically to filter out bit-to-bit skew and present data to the internal data busses in a synchronized manner across the entire width of the data bus.
Figure 3 – Visualizing Inter-Device Skew as a Wavefront
Figure 3 shows that in an XDR channel with 16 devices, with each device providing a 1 bit wide connection to the 16 bit wide data bus. In figure 3, the inter-device skew between device 0 and device 15 is much larger than the inter-device skew between device 0 and device 1. As a result, the Flexphase de-skewing circuitry needs to be deeper for the naively designed long channel configuration as compared to the short channel configuration. Abstractly, figure 3 shows that the problem can be thought of in terms of a “wavefront” that travels from the DRAM devices to the XDR memory controller interface for a given column read data command, and the slope of the wave front increases as the number of devices in the XDR memory system is increased.
To counter the growing slope of the wavefront in the XDR memory system for a long channel (large device count) configuration, system design engineers can choose to purposefully design in skewed data paths for each device or to increase the depth of the Flexphase de-skewing circuitry. That is, system design engineers can choose to increase the path length of bit 0 so that it would have to travel for a longer distance than bit 15. Such a design would reduce the need for the deeper Flexphase de-skewing circuitry, but it could expose the data bus signals to an increasing amount of temperature-dependent jitter, possibly resulting in a less stable platform. As a result, the deeper Flexphase circuitry would be favored as a more stable solution.
To summarize, the XDR memory system in the CELL processor can in all likelihood support as many as 72 XDR DRAM devices. The technical issue at this point in time is whether IBM has fully considered the engineering challenges of supporting 72 devices in the two channel XDR memory systems and validated that configuration (and have the DRAM controller issue sub-column commands, but that is presumably a relatively minor issue). The long channel configuration support minimally requires sufficient depth in the Flexphase de-skewing circuitry to account for the extremes in inter-device data skew with 36 DRAM devices per channel. Additionally, the non-technical factor to the use of XDR memory is the price premium of XDR DRAM devices. The price premium is a significant factor that could deter wide scale adoption of CELL processors with large memory configurations.
One final intriguing note is that in the long channel, 1-bit-wide-data-bus-per-device configuration, a pleasant side effect is that the memory system gains automatic chipkill capability with simple ECC support.
Pages: « Prev 1 2 3 4 5 Next »
Discuss (5 comments)