Use of Intel® MKL in HPCC benchmark

HPCC Application Note

Step 1 - Overview

This guide is intended to help current HPCC users get better benchmark performance by utilizing Intel® Math Kernel Library (Intel® MKL).

HPCC stands for High Performance Computing Challenge benchmark and is actually a suite of benchmarks that measure performance of the CPU, memory subsystem and interconnect. It consists of 7 benchmark tests - HPL (High Performance LINPACK), DGEMM (Double-precision GEneral Matrix-Matrix multiply), STREAM, PTRANS (Parallel TRANSpose, Random Access, FFT (Fast Fourier Tranform and communication bandwidth/latency.

Please find more information on HPCC from: http://icl.cs.utk.edu/hpcc/* .

Version Information

This application note was created to help users who benchmark clusters using HPCC to make use of the latest versions of Intel MKL on Linux platforms on Xeon systems.  Please note that previous versions of MKL may require other steps to successfully compile and link with HPCC.

Step 2 - Downloading HPCC Source Code

The HPCC source code can be downloaded from: http://icl.cs.utk.edu/hpcc/software/index.html*.

Prerequisites

1.     Intel MKL contains highly optimized FFT and also the wrappers for FFTW, which can be obtained from the following options:

•  Download a FREE evaluation version of the Intel MKL product.

•  Download the FREE non-commercial* version of the Intel MKL product.

All of these can be obtained at: Intel® Math Kernel Library product web page.

Intel® MKL is also bundled with the following products

2.     Intel MPI can be obtained from Intel® Cluster Tools. Open source MPI (MPICH2) can be obtained from http://www.mcs.anl.gov/research/projects/mpich2/*.

Step 3 - Configuration

Use the following commands to extract the HPCC tar files from the downloaded hpcc-x.x.x.tar.gz.

$gunzip hpcc-x.x.x.tar.gz
$tar -xvf hpcc-x.x.x.tar

The above will create a directory named hpcc-x.x.x

Make sure that MPI, C++ and FORTRAN compilers are installed and they are in PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and MKL libraries.

Step 4 - Building HPCC

•  Build MPI MKL FFTW library.

Change the directory to <your MKL installation>/interfaces/fftw2x_cdft.

From the fftw2x_cdft directory, run the following command:

$make libintel64 PRECISION=MKL_DOUBLE interface=lp64

Here we are building for Intel64 architecture with Intel MPI (default for Makefile, you may use a different mpi), with Intel compilers, DOUBLE precision and lp64 interface. This will create the MKL MPI FFTW interface library libfftw2x_cdft_DOUBLE_lp64.a in lib/intel64 directory.

Note: Please note that by setting the interface parameter to lp64 we require to build the FFTW MPI wrappers which admit 64-bit parameters in their interface to match the calls from HPCC.  These 64-bit aware wrappers are not to be used with usual applications complying with traditional FFTW interfaces.  Please execute $make to see the full set of options.

•  Build FFTW C wrapper library

Change the directory to <your MKL installation>/interfaces/fftw2xc.

Then build the FFTW wrapper by running the command as below
 

$make libintel64 PRECISION=MKL_DOUBLE


This will create libfftw2xc_intel.a library in <your mkl installation>/lib/intel64 directory

•  Build HPCC

Change directory to hpcc-x.x.x/hpl

Create a Makefile from the existing one, for e.g. Make.intel. You can reuse one from the hpl/setup directory.

Edit Make.intel as follows: modify the LAdir, LAlib lines as below to point to MKL libraries.

LAdir = /opt/intel/mkl/lib/intel64
LAlib = -Wl,--start-group $(LAdir)/libfftw2x_cdft_DOUBLE_lp64.a $(LAdir)/libfftw2xc_intel.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_intel_thread.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a $(LAdir)/libmkl_cdft_core.a -Wl, --end-group -lpthread -lm


Please make sure to following compiler options on the compile line:

-DUSING_FFTW -DMKL_INT=long -DLONG_IS_64BITS

Build HPCC by using
 

$make all arch=intel


This will create an executable with name hpcc in the hpcc-x.x.x directory and a file _hpccinf.txt which is a template input file for hpcc. Rename the file to hpccinf.txt.

Step 5 - Running HPCC

Modify the configuration parameters in hpccinf.txt file.

Run hpcc by executing the following command.

$mpirun -np 4 hpcc


hpccinf.txt is the same as standard hpl input file with a few additional lines. Please refer our HPL application note on tuning parameters in the configuration file.

Appendix A - Performance Results

hpcc benchmark results of Intel Endeavor cluster are available in hpcc website*.

Appendix C - References

Intel Xeon Processor based Servers Homepage


 
For more complete information about compiler optimizations, see our Optimization Notice.

12 comments

Top
hilgeman's picture

Vipin,

When I use MKL 10.3.7.256, StarFFT aborts with the following message:

fftw_die: DftiCompute returned error in fftwnd()

MKL 10.3.0.084 is fine, though. I am using the 11.1.073 compilers.

regards,

-Martin

Vipin Kumar E K (Intel)'s picture

Roger,

The article has been already updated with the a note in the

Step 4 - Building HPCC

• Build MPI MKL FFTW library.

as

Note: Please note that by setting the interface parameter to ilp64 we require to build the FFTW MPI wrappers which admit 64-bit parameters in their interface to match the calls from HPCC. These 64-bit aware wrappers are not to be used with usual applications complying with traditional FFTW interfaces. Please execute $make to see the full set of options.

--Vipin

Vipin Kumar E K (Intel)'s picture

Hi Anna,

May be this article can be of some help.

http://software.intel.com/en-us/articles/64-bit-int-support-on-win64-mkl/

--Vipin

Vipin Kumar E K (Intel)'s picture

Hi Roger,

The Make.intel is attached in the article attachments section above.

The LAinc and CCLAGS are

LAdir = $(MKL)
LAinc = -I$(MKL)/include

CCFLAGS = $(HPL_DEFS) -DASYOUGO -O2 -xSSE4.2 -ip -ansi-alias -fno-alias -DUSING_FFTW -DMKL_INT=long -L$MKL/lib/intel64 -I$MKL/include/fftw -L$MPI/lib64 -DLONG_IS_64BITS -DRA_SANDIA_OPT2 -DHPCC_FFT_235

We are in the process of updating this article to the latest version of Intel MKL, Compilers and MPI.

Thanks,
Vipin

anonymous's picture

Would you have the kindness to send me the parameters file used for the test reported in this paga, please?
Thanks in advance

anonymous's picture

you build fftw_cdft ilp64, but in the link section you pull in lp64 (not ilp64) for everything.
Could you update the instructions for the 12.0 compilers, intel mpi 4.0.1.007.
could you also include the LAinc parameter and CCFLAGS, etc. Maybe you could post your hpl/Make.intel file.
Thanks,
Roger

anonymous's picture

I tried to build HPCC with MKL and msmpi on Windows. I'm using Intel Composer 2011.1.127 and MKL 10.3.1.127 in VS2010. I successfully built fftw2x_cdft_DOUBLE_ilp64.lib and fftw2xc_intel.lib for ilp64 interface.

I link HPCC with

msmpi.lib
fftw2x_cdft_DOUBLE_ilp64.lib
fftw2xc_intel.lib
mkl_blacs_msmpi_ilp64.lib
mkl_cdft_core.lib
mkl_core.lib
mkl_intel_ilp64.lib
mkl_intel_thread.lib

When I use just two compiler flags /DUSING_FFTW /DMKL_ILP64, everything seems to work fine (the only thing that annoys me that StarFFT gets very memory-consuming). I cannot use /DMKL_INT=long /DLONG_IS_64BITS flags, when I build with these flags the code wouldn't start and the output message I get from HPCC is "No 64-bit integer type available". How do I fix it? And will the solution help me with memory consuming issues?

xuzheng97's picture

Hello,

I tested as guided including "Build FFTW C wrapper library" part.
But I still got error as following:

*** glibc detected *** ./hpcc: free(): invalid pointer: 0x0000003ce8352d48 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ce80722ef]
/lib64/libc.so.6(cfree+0x4b)[0x3ce807273b]
./hpcc[0x4cbb0a]

I am using Intel compiler 11.1.072 and corresponding MKL in the compiler directory.
Also MPI is 4.0.0.028.
I tried both HPCC1.4.1 and 1.3.1 with same error

Thanks

andres-more (Intel)'s picture

As a side note, the performance figures of STREAM can be increased in Intel architecture by explicitly using the '-opt-streaming-stores always' option of the Intel C Compiler when compiling the STREAM source.

patirot's picture

Thank you very much!

Pages

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.