SPECjbb2005 is an industry standard benchmark for evaluating server side Java performance and is a licensed product of the Standards for Performance Evaluation Cooperative (SPEC); SPECjbb is a trademark of SPEC. It is loosely derived from TPC-C, although it forgoes the database aspect and focuses on the performance of the middle tier of a three-tier architecture, implemented in Java. Quoting from SPEC’s website:“SPECjbb2005 simulates a wholesale company with warehouses that serve different districts. It mimics customer operations such as placing orders or requesting the status of an existing order, and operations within the company, such as processing orders for delivery, entering customer payments, checking stock levels, and requesting a report on recent activity by a given customer.
The benchmark measures throughput of the underlying Java platform, which is the rate at which business operations are performed per second. It steps through increasing amounts of work, providing a graphical view of scalability. Performance is assessed by two metrics: bops (business operations per second), which measures overall throughput for all of the JVMs in a benchmark run, and bops/JVM, which measures the performance and scaling of a single JVM.
The benchmark exercises the implementations of the JVM (Java Virtual Machine), JIT (Just-In-Time) compiler, garbage collection, threads and some aspects of the operating system. It also measures the performance of CPUs, caches, memory hierarchy and the scalability of shared memory processors (SMPs).”
For more information, please visit www.spec.org/jbb2005. Our measurements were done in mid-November using a single JVM on the Nocona and Bensley systems specified in the introduction. The following command line options were used:
-server -Xms1500m -Xmx1500m -Xss256K -XX:+AggressiveHeap
Each warehouse independently spawns a thread in the benchmark, thus determining the amount of concurrency in the benchmark. Each system has an expected peak number of warehouses (N), which corresponds to the total number of hardware threads supported. The score, measured in Business OPerations per second (BOP/s) is actually an average of the throughputs for N, N+1, N+2…2N warehouses. Due to SPECjbb2005 run rules, we will not be giving actual numbers for the Bensley system.
Figure 5 – SPECjbb2005 Relative Throughput
Figure 5 above shows the throughput graphs for three systems: the Nocona, the Bensley (Bensley 4GB) and the Bensley with 1 512MB FB-DIMM per channel (Bensley 2GB). The data points circled in red are those used for calculating actual scores. The Nocona system averaged 21611 BOP/s over three runs, with a standard deviation of 289. Both Bensley configurations score about 2.1 times the throughput of the Nocona system in BOP/s.