Introduction
Our introduction to Sandy Bridge-EP covers the high level details Xeon E5 series. In this follow-on article, we explore the performance and power efficiency of Intel’s new server platform and discuss the implications for the broader server market.
Sandy Bridge-EP comes with a new platform, codenamed Romley. The integrated PCI-E, additional channel of memory and platform level power management are fundamentally different than for Westmere-EP, requiring a break in compatibility. For our review, two separate server systems were evaluated for performance and power efficiency.
System Configuration
Intel’s Grizzly Pass reference motherboard houses two Xeon E5-2690 processors, with a base frequency of 2.9GHz. Each processor contains 8 Sandy Bridge cores, 20MB L3 cache, 2 QPI links operating at 8GT/s, 40 lanes of 8GT/s PCI-E 3.0 and 4 channels of 1.6GT/s DDR3 with a 135W TDP rating. Both QPI links are connected between the two processors, so the communication bandwidth is actually 80GB/s. The enhanced power management can run the processor at 3.3GHz with 6-8 cores active, which is the region of interest for most of our benchmarks, although the peak frequency is an impressive 3.8GHz for a single core.
Westmere-EP is represented by two X5670 CPUs running at 2.93GHz; each contains 6 cores, 12 threads and a 12MB L3 cache, with a TDP of 95W. With the dynamic voltage and frequency scaling (DVFS) that Intel calls Turbo Boost, the X5670 can reach 3.2GHz with 3-6 cores active, and 3.33GHz with 1-2 cores active. The Intel 5520 chipset has 36 lanes of 5GT/s PCI-E 2.0 and a TDP of 27W, although the actual power consumed should be far less. Since a QPI link is used to connect each processor to the northbridge, there is only 32GB/s for communication between the processors.
Chart 1. System Configurations
The systems were configured as shown above in Chart 1. The same DIMMs were used in both systems, although in the case of the Westmere-EP, it is running at a lower frequency due to memory controller limitations. The Sandy Bridge-EP system came equipped with two power supplies for redundancy, but the second was not used and is omitted from the chart above.
The newer Sandy Bridge-EP system is a larger 2U chassis, which has implications for cooling and power consumption. All things being equal, it takes more power to efficiently operate and cool a smaller server, but it is rather difficult to estimate the impact.
The OS power management policy was set to balanced, which actually disables Turbo-mode for Westmere-EP. This was a conscious decision, as the hardware power management is overly aggressive which leads to efficiency losses for many workloads. While this accurately reflects real world conditions, it is important to note in the context of the review. One of the improvements in the Sandy Bridge-EP is more intelligent power management that will be enabled by the OS.
Power measurements (watts at the wall, taken with a Watts Up Pro meter) were logged on the system we were testing at 1 second intervals over a USB connection. We measured a negligible impact on performance from this logging, and discussions with some of the architects of SPECpower have confirmed this behavior. However, logging on the system under test does have a Heisenberg-like impact on power consumption. The system will never truly quiesce into a full idle state (where all CPUs can shut down), while it is logging power measurements. Rather it will reach an ‘active idle’ state. Active idle for the Sandy Bridge-EP system is 109W, substantially less than the 162W measured for the Westmere-EP system. The difference reflects the more advanced processor and platform power management in Sandy Bridge-EP.
The benchmarks for this review include:
- Euler3d
- SPECpower_ssj2008
- SPECcpu_2006
Performance for Euler3d is the average of three runs, while SPECpower_ssj2008 and SPECcpu_2006 which both use a single run. Power measurements are taken from a single run, rather than averaged across multiple runs.
Discuss (15 comments)