HPCC Application Note
Step 1 - Overview
This guide is intended to help current HPCC users get better benchmark performance by utilizing Intel® Math Kernel Library (Intel® MKL).
HPCC stands for High Performance Computing Challenge benchmark and is actually a suite of benchmarks that measure performance of the CPU, memory subsystem and interconnect. It consists of 7 benchmark tests - HPL (High Performance LINPACK), DGEMM (Double-precision GEneral Matrix-Matrix multiply), STREAM, PTRANS (Parallel TRANSpose, Random Access, FFT (Fast Fourier Tranform and communication bandwidth/latency.
Please find more information on HPCC from: http://icl.cs.utk.edu/hpcc/* .
This application note was created to help users who benchmark clusters using HPCC to make use of the latest versions of Intel MKL on Linux platforms on Xeon systems. Please note that previous versions of MKL may require other steps to successfully compile and link with HPCC.
Step 2 - Downloading HPCC Source Code
The HPCC source code can be downloaded from: http://icl.cs.utk.edu/hpcc/software/index.html*.
1. Intel MKL contains highly optimized FFT and also the wrappers for FFTW, which can be obtained from the following options:
• Download a FREE evaluation version of the Intel MKL product.
• Download the FREE non-commercial* version of the Intel MKL product.
All of these can be obtained at: Intel® Math Kernel Library product web page.
Intel® MKL is also bundled with the following products
2. Intel MPI can be obtained from Intel® Cluster Tools. Open source MPI (MPICH2) can be obtained from http://www.mcs.anl.gov/research/projects/mpich2/*.
Step 3 - Configuration
Use the following commands to extract the HPCC tar files from the downloaded hpcc-x.x.x.tar.gz.
$gunzip hpcc-x.x.x.tar.gz $tar -xvf hpcc-x.x.x.tar
The above will create a directory named hpcc-x.x.x
Make sure that MPI, C++ and FORTRAN compilers are installed and they are in PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and MKL libraries.
Step 4 - Building HPCC
• Build MPI MKL FFTW library.
Change the directory to <your MKL installation>/interfaces/fftw2x_cdft.
From the fftw2x_cdft directory, run the following command:
$make libintel64 PRECISION=MKL_DOUBLE interface=lp64
Here we are building for Intel64 architecture with Intel MPI (default for Makefile, you may use a different mpi), with Intel compilers, DOUBLE precision and lp64 interface. This will create the MKL MPI FFTW interface library libfftw2x_cdft_DOUBLE_lp64.a in lib/intel64 directory.
Note: Please note that by setting the interface parameter to lp64 we require to build the FFTW MPI wrappers which admit 64-bit parameters in their interface to match the calls from HPCC. These 64-bit aware wrappers are not to be used with usual applications complying with traditional FFTW interfaces. Please execute $make to see the full set of options.
• Build FFTW C wrapper library
Change the directory to <your MKL installation>/interfaces/fftw2xc.
Then build the FFTW wrapper by running the command as below
$make libintel64 PRECISION=MKL_DOUBLE
This will create libfftw2xc_intel.a library in <your mkl installation>/lib/intel64 directory
• Build HPCC
Change directory to hpcc-x.x.x/hpl
Create a Makefile from the existing one, for e.g. Make.intel. You can reuse one from the hpl/setup directory.
Edit Make.intel as follows: modify the LAdir, LAlib lines as below to point to MKL libraries.
LAdir = /opt/intel/mkl/lib/intel64 LAlib = -Wl,--start-group $(LAdir)/libfftw2x_cdft_DOUBLE_lp64.a $(LAdir)/libfftw2xc_intel.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_intel_thread.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a $(LAdir)/libmkl_cdft_core.a -Wl, --end-group -lpthread -lm
Please make sure to following compiler options on the compile line:
-DUSING_FFTW -DMKL_INT=long -DLONG_IS_64BITS
Build HPCC by using
$make all arch=intel
This will create an executable with name hpcc in the hpcc-x.x.x directory and a file _hpccinf.txt which is a template input file for hpcc. Rename the file to hpccinf.txt.
Step 5 - Running HPCC
Modify the configuration parameters in hpccinf.txt file.
Run hpcc by executing the following command.
$mpirun -np 4 hpcc
hpccinf.txt is the same as standard hpl input file with a few additional lines. Please refer our HPL application note on tuning parameters in the configuration file.
Appendix A - Performance Results
hpcc benchmark results of Intel Endeavor cluster are available in hpcc website*.
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804