The scalability port has also been significantly improved over the EXA2. Like the I/O subsystem, the scalability port also uses InfiniBand electricals. The X3 provides three connections to other quads and each link provides upto 6.4GB/s of bandwidth and, like the I/O subsystem, hot fail-over and 32 bit CRC, but no ECC. Remote latency also dropped from 735ns to 222ns, which is only twice local latency, a very impressive accomplishment. These figures are for round trip access with a single hop across the scalability port. Two hops will be introduced for systems with more than 4 nodes. Incidentally, quads can be connected with up to two scalability ports. This is only possible for 8P configurations, but using two scalability ports means that a failure in one of the ports can be tolerated, and the available inter-node bandwidth would be doubled to 12.8GB/s. Unsurprisingly, the RAS benefits are seen as the more important contribution, although it is unclear whether this is because the additional bandwidth has marginal performance impact or because RAS is simply more critical for IBM’s target market, or possibly both.
Now having discussed all the major subsystems of the X3 chipset, let’s take a look at the die for the Hurricane controller, shown below in Figure 5:
Figure 5 – Die Micrograph of X3 Scalability Controller, based on an IBM presentation
Discuss (13 comments)