Last year in mid-December, SPECpower_ssj2008, the first industry standard power efficiency benchmark, was released. The benchmark effort started out in response to the intense focus on, and need for clarity in power consumption within data centers. While Google and Microsoft have teams of engineers dedicated to acquiring or designing the most efficient servers for their needs, most companies do not have the resources necessary to create measurement software, hence they are forced to rely on vendors or industry standard benchmarks.
When power first became a significant issue in the data center, many vendors responded by creating their own metrics and benchmarks for power efficiency. These were unreliable, unscientific, unsystematic and typically formulated in such a way to provide a vendor’s marketing department with ammunition, rather than to actually help end-users; for example, Sun advocated the “SWaP” metric, which was invented with little regard for what it actually meant (ironically, Sun had some of the most power optimized products, courtesy of Niagara). Perhaps the best explanation is that the first few tries at ‘measuring’ or ‘benchmarking’ power resulted in the moral equivalent of Dhrystone MIPS; well intentioned, but generally useless (or worse, misleading) quantifications. In early 2006, the SPECpower working group was formed, with the goal of creating useful and valuable energy efficiency benchmarks for the industry at large in the same way that SPEC CPU was used to create meaningful performance benchmarks starting in 1989.
The release of SPECpower_ssj2008 is an innovative step forward for the IT industry as whole. It establishes a reasonably level playing field for end-users, manufacturers and other participants to judge the efficiency of various systems that are available. By creating a level playing field, it also gives engineers and architects a very clear target to aim for when designing future products, which should result in substantially more energy efficient products.
SPECpower is probably the most complex benchmark that SPEC has undertaken to date, and is actually two significant projects. First and foremost, SPECpower is a framework that measures power efficiency with a workload agnostic methodology and architecture. SPECpower_ssj2008 itself measures power efficiency and performance for server-side Java and is an implementation of this framework that uses a modified version of the SPECjbb2005 workload.