The I/O for the X3 has also been revamped; although some of the differences are not readily apparent from block diagrams. The I/O system actually uses InfiniBand electricals (readily available from your local branch of IBM Microelectric) to connect to a pair of 64 bit PCI-X 2.0 controllers. Each PCI-X bus operates at the full 266MHz, providing a total of 12GB/s in bandwidth across 6 slots, triple that of the EXA2 chipset. The controller uses 32 bit cyclical redundancy checks (CRC) for error detection purposes, but no ECC and supports hot-adds and swaps. While PCI-X is the current I/O technology, IBM may upgrade the X3 to support PCI Express x8 in the future.
An IOMMU by Another Name
While the bandwidth is certainly impressive, the most distinguishing I/O features of the x3 are not highly visible, in particular the I/O Memory Management Unit (IOMMU). Most I/O devices work with memory through a process known as DMA (Direct Memory Access), which bypasses the CPU. As a result, DMAs usually work with physical memory, because virtual address translations are performed in the CPU using the TLBs and page tables. IOMMUs work with DMA capable devices and perform virtual address translation for them, so that the devices can bypass the CPU and perform DMAs. IOMMUs are hardly new to IBM; IBM first patented the concept in 1971, although it was not implemented until the debut of PR/SM on the IBM 3090 in 1985. Based on this history, it should be little surprise that IOMMUs are standard for IBM servers, although they are referred to as Translation Control Entries (TCEs) in IBM jargon.
IBM re-used the IOMMU from the p/i/zSeries systems for the X3. This IOMMU theoretically supports all the features present in the p/i/zSeries IOMMUs. It supports a 4GB I/O address space and requires hypervisor privileges to modify the mapping tables. However, the catch is that it requires BIOS and operating system support to function. It is quite possible that in the future, Linux or Windows will have the appropriate capabilities, but they do not currently. Another alternative, one that is more likely in the near future, is that a hypervisor (or Virtual Machine Monitor/VMM) will be released that can take advantage of the IOMMU. The obvious candidates would be VMWare, and the Xen project. A hypervisor that utilizes the IOMMU would significantly improve the RAS of the X3’s I/O subsystem, and would be a welcome improvement to the x86 server ecosystem.
Discuss (13 comments)