HPC Application Performance on ESX 4.1: Stream

Recently VMware has seen increased interest in migrating High Performance Computing (HPC)
applications to virtualized environments. This is due to the many advantages
virtualization brings to HPC, including consolidation, support for heterogeneous
OSes, ease of application development, security, job
migration, and cloud computing (all described href="http://feeds.vmware.com/l?s=_wordpress&r=wordpress&he=68747470253341253246253246636f6d6d756e69746965732e766d776172652e636f6d253246636f6d6d756e69747925324663746f253246686967682d706572666f726d616e6365&i=727373696e3a7461673a747970657061642e636f6d2c323030333a706f73742d36613030643833343163333238313533656630313333663366653863346339373062">here). Currently some subset of HPC
applications virtualize well from a performance perspective. Our long-term goal
is to extend this to all HPC apps, realizing that large-scale apps with the
lowest latency and highest bandwidth requirements will be the most challenging.
Users who run HPC apps are traditionally very sensitive to performance overhead,
so it is important to quantify the performance cost of virtualization and
properly weigh it against the advantages. Compared to commercial apps
(databases, web servers, and so on), which are VMware’s bread-and-butter, HPC
apps place their own set of requirements on the platform
(OS/hypervisor/hardware) in order to execute well. Two common ones are
low-latency networking (since a single app is often spread across a cluster of
machines) and high memory bandwidth. This article is the first in a series that
will explore these and other HPC performance subjects. Our goal will always be
to determine what works, what doesn’t, and how to get more of the former. The
benchmark reported on here is href="http://feeds.vmware.com/l?s=_wordpress&r=wordpress&he=687474702533412532462532467777772e63732e76697267696e69612e65647525324673747265616d2532467265662e68746d6c&i=727373696e3a7461673a747970657061642e636f6d2c323030333a706f73742d36613030643833343163333238313533656630313333663366653863346339373062">Stream, which is a standard tool designed
to measure memory bandwidth. It is a “worst case” micro-benchmark; real
applications will not achieve higher memory bandwidth.

Configuration

All tests were performed on an HP DL380 with two Intel X5570 processors, 48 GB memory (12
× 4 GB DIMMs), and four 1-GbE NICs (Intel Pro/1000 PT Quad Port Server Adapter)
connected to a switch. Guest and native OS is RHEL 5.5 x86_64. Hyper-threading
is enabled in the BIOS, so 16 logical processors are available. Processors and
memory are split between two NUMA nodes. A pre-GA lab version of ESX 4.1 was
used, build 254859.

Test Results

The OpenMP version of Stream is used. It is built using a
compiler switch as follows:

gcc -O2 -fopenmp stream.c -o stream

The number of simultaneous threads is controlled by an environment
variable:

export OMP_NUM_THREADS=8

The array size (N) and number of iterations (NTIMES) are hard-wired in the code as
N=10⁸ (for a single machine) and NTIMES=40. The large array size
ensures that the processor cache provides little or no benefit. Stream reports
maximum memory bandwidth performance in MB/sec for four tests: copy, scale, add,
and triad (see the above link for descriptions of these). M stands for 1
million, not 2²⁰. Here are the native results, as a function of the
number of threads:

Table 1. Native memory bandwidth, MB/s

cellSpacing=0 cellPadding=0 border=1>