HPCC Application Note

Step 1 - Overview

This guide is intended to help current HPCC users get better benchmark performance by utilizing Intel® Math Kernel Library (Intel® MKL).

HPCC stands for High Performance Computing Challenge benchmark and is actually a suite of benchmarks that measure performance of the CPU, memory subsystem and interconnect. It consists of 7 benchmark tests - HPL (High Performance LINPACK), DGEMM (Double-precision GEneral Matrix-Matrix multiply), STREAM, PTRANS (Parallel TRANSpose, Random Access, FFT (Fast Fourier Tranform and communication bandwidth/latency.

Please find more information on HPCC from: http://icl.cs.utk.edu/hpcc/* .

Version Information

This application note was created to help users who benchmark clusters using HPCC to make use of the latest versions of Intel MKL on Linux platforms on Xeon systems.  Please note that previous versions of MKL may require other steps to successfully compile and link with HPCC.

Step 2 - Downloading HPCC Source Code

The HPCC source code can be downloaded from: http://icl.cs.utk.edu/hpcc/software/index.html*.

Prerequisites

1.     Intel MKL contains highly optimized FFT and also the wrappers for FFTW, which can be obtained from the following options:

•  Download a FREE evaluation version of the Intel MKL product.

•  Download the FREE non-commercial* version of the Intel MKL product.

All of these can be obtained at: Intel® Math Kernel Library product web page.

Intel® MKL is also bundled with the following products

• Intel® Parallel Studio XE 2013
• Intel® Composer XE 2013
• Intel® Cluster Studio 2013

2.     Intel MPI can be obtained from Intel® Cluster Tools. Open source MPI (MPICH2) can be obtained from http://www.mcs.anl.gov/research/projects/mpich2/*.

Step 3 - Configuration

Use the following commands to extract the HPCC tar files from the downloaded hpcc-x.x.x.tar.gz.

[shell]$gunzip hpcc-x.x.x.tar.gz
$tar -xvf hpcc-x.x.x.tar
[/shell]
The above will create a directory named hpcc-x.x.x

Make sure that MPI, C++ and FORTRAN compilers are installed and they are in PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and MKL libraries.

Step 4 - Building HPCC

•  Build MPI MKL FFTW library.

Change the directory to <your MKL installation>/interfaces/fftw2x_cdft.

From the fftw2x_cdft directory, run the following command:

[shell]$make libintel64 PRECISION=MKL_DOUBLE interface=ilp64[/shell]
Here we are building for Intel64 architecture with Intel MPI (default for Makefile, you may use a different mpi), with Intel compilers, DOUBLE precision and ilp64 interface. This will create the MKL MPI FFTW interface library libfftw2x_cdft_DOUBLE_ilp64.a in lib/intel64 directory.

Note: Please note that by setting the interface parameter to ilp64 we require to build the FFTW MPI wrappers which admit 64-bit parameters in their interface to match the calls from HPCC.  These 64-bit aware wrappers are not to be used with usual applications complying with traditional FFTW interfaces.  Please execute $make to see the full set of options.

•  Build FFTW C wrapper library

Change the directory to <your MKL installation>/interfaces/fftw2xc.

Then build the FFTWC wrapper by running the command as below

[shell]$make libintel64 PRECISION=MKL_DOUBLE[/shell]
This will create libfftw2xc_intel.a library in <your mkl installation>/lib/intel64 directory

•  Build HPCC

Change directory to hpcc-x.x.x/hpl

Create a Makefile from the existing one, for e.g. Make.intel. You can reuse one from the hpl/setup directory.

Edit Make.intel as follows: modify the LAdir, LAlib lines as below to point to MKL libraries.
[shell]LAdir = /opt/intel/mkl/lib/intel64

LAlib = -Wl,--start-group $(LAdir)/libfftw2x_cdft_DOUBLE_lp64.a $(LAdir)/libfftw2xc_intel.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_intel_thread.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a $(LAdir)/libmkl_cdft_core.a -Wl, --end-group -lpthread -lm
[/shell]
Please make sure to following compiler options on the compile line:

[shell]-DUSING_FFTW -DMKL_INT=long -DLONG_IS_64BITS[/shell]
Build HPCC by using

[shell]$make all arch=intel[/shell]
This will create an executable with name hpcc in the hpcc-x.x.x directory and a file _hpccinf.txt which is a template input file for hpcc. Rename the file to hpccinf.txt.

Step 5 - Running HPCC

Modify the configuration parameters in hpccinf.txt file.

Run hpcc by executing the following command.

[shell]$mpirun -np 4 hpcc[/shell]
hpccinf.txt is the same as standard hpl input file with a few additional lines. Please refer our HPL application note on tuning parameters in the configuration file.

Appendix A - Performance Results

Below are the hpcc benchmark results of Intel Endeavor cluster which can also be found in hpcc website*.

Appendix C - References

• Intel Xeon Processor based Servers Homepage