NAG ports a range of its software to both Intel®/Windows* and Intel®/Linux* platforms. NAG and Intel have been working closely together since the introduction of the Pentium chip, producing early ports of both libraries and visualization software for various launch events and demonstration CDs.

NAG is working closely with Intel to have early release ports of NAG products. Intel provides a good range of its latest hardware to NAG to ensure that our customers are well supported. In an Intel case study called, "Optimized Number Crunching" http://software.intel.com/file/6278.

It points out Numerical Algorithms Group (NAG) authors high-quality numerical software libraries in order to deliver high performance linear algebra solutions to its customers. For the best solutions running on the latest Intel® processors, NAG libraries use the Intel® Math Kernel Library (Intel® MKL). The case study shows that using NAG & Intel MKL for DGEMM (BLAS), DPOTRF(LAPACK), DPOTRF(LAPACK), and DGEQRF(LAPACK) can achieve up to 3.5 times increase over using NAG library alone.

List of NAG products available on Intel® processor-based systems

• (LUX) x86-32 Linux PC/Unix
• (DLL) Intel-32 Windows PC/Windows
• (NTI) Intel-32 Windows PC/Windows
• (W32) x86-32 Windows PC/Windows
• (W64) Itanium Windows64 PC/Windows

See http://www.nag.com/doc/inun/fl22.html for the currently available implementation for the NAG Fortran Library, Mark 22.

List of NAG Installer’s Notes which include Intel® MKL

EM64T, Linux 64, Intel FORTRAN, Double Precision Installer's Note

Fortran compiler: Intel® Fortran compiler for Intel® 64, Version 10.1 and compatible.

Vendor library: Intel MKL version 10.1.1.019

The NAG library has been tested with Intel MKL version 10.1.1.019 and is included as a part of the product (see contents).

How to link Intel® MKL in NAG library

The following sections are extracted from NAG SMP Library, Mark 22 Users’ Note for Linux 64 (Intel® 64 / AMD64), Intel Fortran, Double Precision. Please refer to this document for details.

1. Libraries NOT in the linker search path In this section we assume that the library has been installed in the directory [INSTALL_DIR]. By default, [INSTALL_DIR] is /opt/NAG/fll6i22dcl or /usr/local/NAG/fll6i22dcl depending on your system (see Installer's Note (in.html)); however it could have been changed by the installer. To identify [INSTALL_DIR] for this installation:
   • if /opt/NAG/fll6i22dcl exists, then this is [INSTALL_DIR]
   • if /usr/local/NAG/fll6i22dcl exists, then this is [INSTALL_DIR]
   • consult the installer for the directory [INSTALL_DIR]

   To use the NAG Fortran Library and the supplied MKL libraries, you may link in the following manner:

   ifort driver.f [INSTALL_DIR]/lib/libnag_mkl.a \
   -Wl,--start-group $[INSTALL_DIR]/mkl_em64t/libmkl_intel_lp64.a \
$[INSTALL_DIR]/mkl_em64t/libmkl_intel_thread.a \
$[INSTALL_DIR]/mkl_em64t/libmkl_core.a -Wl,--end-group -lguide -lpthread
   
   where driver.f is your application program; or
   
   ifort driver.f [INSTALL_DIR]/lib/libnag_mkl.so -L[INSTALL_DIR]/mkl_em64t \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_lapack -lmkl -lmkl_core \
-lguide -lpthread
   
   if the shareable library is required.

   The environment variable LD_LIBRARY_PATH must be set to include [INSTALL_DIR]/mkl_em64t. In addition if your application uses the NAG shareable library then the environment variable LD_LIBRARY_PATH must contain [INSTALL_DIR]/lib. If for example, LD_LIBRARY_PATH may be extended as follows to allow run time linkage.

   In the C shell, type:
   
   setenv LD_LIBRARY_PATH [INSTALL_DIR]/lib:[INSTALL_DIR]/mkl_em64t:${LD_LIBRARY_PATH}
   
   to extend LD_LIBRARY_PATH.

   In the Bourne shell, type:
   
   LD_LIBRARY_PATH=[INSTALL_DIR]/lib:[INSTALL_DIR]/mkl_em64t:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
   
   to extend LD_LIBRARY_PATH.


2. Libraries in the linker search path

   In this section we assume that the NAG compiled libraries and the supplied Intel MKL libraries are installed in, or are pointed at by symbolic links from a directory in the search path of the linker, such as /usr/lib.

   To use the NAG Fortran Library and the supplied Intel MKL libraries in this context, you may link in the following manner:

   ifort driver.f -lnag_mkl -lmkl_intel_lp64 -lmkl_intel_thread \
-lmkl_lapack -lmkl -lmkl_core -lguide -lpthread

   This will usually link to the shareable library in preference to the static library if both the libraries are at the same location.

   To use the static library libnagsmp.a you need the -Bstatic compiler flag:

   ifort -Bstatic driver.f -lnag_mkl -Wl,--start-group -lmkl_intel_lp64 \
-lmkl_intel_thread -lmkl_core -Wl,--end-group -Bdynamic -lguide -lpthread

   The environment variable LD_LIBRARY_PATH must be set to allow runtime linkage.

   If for example your application has been linked with the NAG shareable library then the environment variable LD_LIBRARY_PATH must be extended as follows, to allow run time linkage.

   In the C shell, type:

   setenv LD_LIBRARY_PATH [INSTALL_DIR]/lib:[INSTALL_DIR]/mkl_em64t:${LD_LIBRARY_PATH}

   to extend LD_LIBRARY_PATH.

   In the Bourne shell, type:

   LD_LIBRARY_PATH=[INSTALL_DIR]/lib:[INSTALL_DIR]/mkl_em64t:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH

   to extend LD_LIBRARY_PATH.

Other NAG and Intel MKL products information

NAG Software Downloads - SMP Library: http://www.nag.com/downloads/fsdownloads.asp

The Intel® MKL Release Notes, Reference Manual, and User's Guide can all be found online: http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation/.

For additional Intel software products information visit: http://www.intel.com/software/products/

Intel, the Intel logo, Itanium, Pentium, Intel Xeon and VTune are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Operating System:

Red Hat* Linux, Red Hat* Desktop Linux* 3, Red Hat* Enterprise Linux Desktop 4, Red Hat* Desktop 3 Update 4, Windows* XP Professional x64 Edition, Windows Server* 2003 Standard x64 Edition, Windows Server* 2003 Enterprise x64 Edition, Red Hat* Enterprise Linux Desktop 3 Update 3, Red Hat* Enterprise Linux Desktop 3 Update 4, Red Hat* Enterprise Linux Desktop 3 Update 5, Red Hat* Enterprise Linux Desktop 4 Update 1, Red Hat* Enterprise Linux 2.1, SUSE* Linux 9.1, SUSE* Linux Enterprise Server 8.0, SUSE* Linux Ente rprise Server 9.0, Red Hat* Enterprise Linux 4.0, Windows* Storage Server, Redhat* Desktop 3 Update 5, Redhat* Desktop 3 Update 6, Redhat* Desktop 3 Update 7, Redhat* Desktop 4 Update 2, Redhat* Desktop 4 Update 3, Redhat* Desktop 4 Update 4, SuSE* Linux* Enterprise* Desktop 10, SUSE* Linux Enterprise Server 10, Windows Vista* 64, Windows Vista* Starter, 32-bit version, Windows Vista* Home Basic, 32-bit version, Windows Vista* Home Premium, 32-bit version, Windows Vista* Business, 32-bit version, Windows Vista* Enterprise, 32-bit version, Windows Vista* Ultimate, 32-bit version, Windows Vista* Home Basic, 64-bit version, Windows Vista* Home Premium, 64-bit version, Windows Vista* Business, 64-bit version, Windows Vista* Enterprise, 64-bit version, Windows Vista* Ultimate, 64-bit version, Windows Vista*, Windows Vista* 32, Windows Server* 2003 for Itanium-based Systems, Windows* XP Starter Edition, Red Hat* Enterprise Linux 5.0, Windows* XP 64-Bit Edition, Windows* XP Professional, Windows* XP Home Edition, Red Hat* Linux 6.2, Red Hat* Linux 6.2 SBE2, Red Hat* Linux 7.0, Red Hat* Linux 7.1, Red Hat* Linux 7.2, Red Hat* Linux 7.3, SUSE* Linux 7.3, SUSE* Linux 8.0, SUSE* Linux 8.1, Red Hat* Linux 8.0, SUSE* Linux 7.2, SUSE* Linux 7.1, SUSE* Linux 7.0, SUSE* Linux, Red Hat* Linux Advanced Server 2.x, Windows* XP Tablet PC Edition, Windows Server* 2003, Windows* XP Media Center Edition, Red Hat* Linux 9.0, Red Hat* Enterprise Linux 3.0, SUSE* Linux* 8.2, Windows Server* 2003 Standard Edition, Red Hat* Linux Advanced Server 3.x, SUSE* Linux* 9.x, Windows* XP 64-Bit Edition Version 2003

• Release Notes
  • Intel® Integrated Performance Primitives for Windows* Release Notes (PDF)
  • Intel® Integrated Performance Primitives for Linux* Release Notes (PDF)
  • Cryptography for Intel® Integrated Performance Primitives for Windows* Release Notes (HTML)
  • Cryptography for Intel® Integrated Performance Primitives for Linux* Release Notes (HTML)


• Intel® Integrated Performance Primitives Reference Manual
  • Volume 1: Signal Processing
    (HTML) (PDF) (CHM)
  • Volume 2: Image and Video Processing
    (HTML) (PDF) (CHM)
  • Volume 3: Small Matrices and Realistic Rendering
    (HTML) (PDF) (CHM)
  • Volume 4: Cryptography
    (HTML) (PDF) (CHM)


• User's Guides
  • Intel® Integrated Performance Primitives for Windows* OS on IA-32 Architecture User’s Guide
    (HTML) (PDF)
  • Intel® Integrated Performance Primitives for Windows* OS on Intel® 64 Architecture User’s Guide
    (HTML) (PDF)
  • Intel® Integrated Performance Primitives for Windows* OS on IA-64 Architecture User’s Guide
    (PDF)
  • Intel® Integrated Performance Primitives for Linux* OS on IA-32 Architecture User’s Guide
    (PDF)

This guide is intended to help current HPL users get better benchmark performance by utilizing BLAS from the Intel® Math Kernel Library (Intel® MKL).

HPL (High Performance LINPACK), an industry standard benchmark for HPC, is a software package that solves a (random) dense linear system in double precision (64 bits) arithmetic on distributed-memory computers.

We will be explaining 3 ways in this note to get the HPL running.

1. Using Intel optimized HPL binary directly (mp_inpack)
2. Building and using HPL from source provided in MKL package
3. Building and using open source HPL by linking with MKL

Version Information

This application note was created to help users who benchmark clusters using HPL to make use of the latest versions of Intel MKL on Linux platforms. Specifically we'll address Intel MKL version 9.1.

Open source HPL can be downloaded from http://www.netlib.org/benchmark/hpl †
If you have installed MKL, HPL is included in MKL and can be found at

<MKL installation dir>/benchmarks/mp_linpack

Prerequisites

• BLAS (Basic Linear Algebra Subprograms)
  
  BLAS DGEMM is the core high performance routine exercised by HPL. Intel® MKL BLAS is highly optimized for maximum performance on Intel® Xeon® processor-based and Itanium® 2-based systems.
  
  BLAS from MKL can be obtained from the following options
• Download a FREE evaluation version of the Intel MKL product
• Download the FREE non-commercial version of the Intel MKL product
• Download one of our FREE Intel Optimized LINPACK benchmark packages
• All of these can be obtained at http://www.intel.com/cd/software/products/asmo-na/eng/307757.htm

FREE Intel Optimized LINPACK Benchmark packages

• The Intel MKL team provides FREE Intel Optimized LINPACK Benchmark packages that are binary implementations of the LINPACK benchmarks which include Intel MKL BLAS. Not only are these SMP and Distributed Memory packages free, they are also much easier to use than HPL (no compilation needed, just run the binaries). We highly recommend HPL users consider switching from HPL to the Free Intel Optimized LINPACK benchmark packages.

Intel® MPI

• Intel MPI can be obtained from http://www.intel.com/cd/software/products/asmo-na/eng/244171.htm

Open source MPI (MPICH2)

• Open source MPI (MPICH2)can be obtained from http://www-unix. mcs.anl.gov/mpi/mpich/ †

Extract the tar file

Use the following commands to extract the tar file from the downloaded hpl.tar.gz file

• Makefile Creation
  • If you are using HPL from Intel MKL you can then use the makefile Make.em64t directly. The top-level directory, in this case will be <your mkl installation>/benchmarks/mp_linpack.
  • If you are using open source HPL:
    
    Create a file Make.<arch> in the top-level directory. For this purpose, you may want to re-use one contained in the setup directory (hpl\setup\). Let us use Make.Linux_PII_CBLAS. This file essentially contains the compilers and libraries with their paths to be used.
• Copy this file:

$cp hpl\setup\Make.Linux_PII_CBLAS hpl


• Rename this file:

$mv Make.Linux_PII_CBLAS Make.em64t.


• Change value of ARCH to em64t (Whichever the value, you have given for <arch>)
• Point to your MPI library
• Point to the math library, MKL
• Extract the package and run the script for your platform.
  In this case, for example, for Xeon 64 bit.
• Go to the directory where the executable is built.
  e.g: For the test run of hpl, use the following commands.
• Create a machines file with node names.
• For e.g. machines files contains names as
  

  front-end-0
  compute-0
  compute-1
  …………………
  …………………
  compute-128
  

  Running with the machines file.
• Intel® Cluster Tools
• Intel Xeon Processor- and Itanium 2-based Servers Homepage

If you have the MKL 9.1 installations, please skip step of extracting HPL. If you downloaded hpl.tar.gz (from netlib) please follow instructions for extracting HPL.

$gunzip hpl.tar.gz
$tar -xvf hpl.tar.

This will create an hpl directory, which we call below the top-level directory.

$cp hpl\setup\Make.Linux_PII_CBLAS hpl

$mv Make.Linux_PII_CBLAS Make.em64t.

Make sure that Intel® C++ and FORTRAN compilers are installed and they are in PATH, also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), MPI, and MKL libraries.

The steps below will explain the steps for building HPL

Edit Make.em64t

# -------------------------------------------------
# - Platform identifier ---------------------------
# -------------------------------------------------
#
ARCH = em64t

MPdir = /opt/intel/mpi
MPinc = -I$(MPdir)/include
MPlib = $(MPdir)/lib/libmpi.a

If you are using gnu MPI (MPICH2), it would be libmpich.a instead of libmpi.a

It is advisable to use Intel MPI for better performance.

LAdir = /opt/intel/mk/9.1/em64t/lib
LAinc = /opt/intel/mkl/9.1/include
LAlib = $(LAdir)/libmkl_em64t.a $(LAdir)/libguide.a –lpthread

To build the executable use "make arch=<arch>". This should create an executable in the bin/<arch> directory called xhpl.

In our example, execute

$make arch=em64t

This creates the executable file bin/em64t/xhpl. It also creates a HPL configuration file HPL.dat.

Typically, scripts are needed to be run, and perhaps portions of the readme file should be reprinted.

Also list the compiler command line syntax, etc.

Case 1: If you have downloaded Intel® Optimized linpack

$runme_xeon64

Please refer the lpk_notes_lin.htm provided with this package for more details

Case 2 & 3: If you have built the hpl from the mkl package or open source hpl

$cd bin/<arch>
$mpirun -np 4 xhpl

$mpirun –np 8 –nodes 4 –machinefile machines xhpl

Please refer MPI documentation for various other arguments, which you can use.

Most of the performance parameters can be tuned, by modifying the input file bin/HPL.dat. See the file TUNING in the top-level directory for more information.

Note: If you use Intel Optimized linpack, you have to change the input files provided with that package (not HPL.dat). For e.g: lininput_xeon64. You can refer the extended help xhelp.lpk for more info in modifying the input file.

Problem size (N): Your problem size should be the largest to fit in the memory to get best performance. For e.g.: If you have 10 nodes with 1 GB RAM, total memory is 10GB. i.e. nearly 1342 M double precision elements. Square root of that number is 36635. You need to leave some memory for Operating System and other things. As a rule of thumb, 80% of the total memory will be a starting point for problem size (So, in this case, say, 33000). If the problem size is too large, it is swapped out, and the performance will degrade.

Block Size (NB): HPL uses the block size NB for the data distribution as well as for the computational granularity. A very small NB will limit computational performance because no data reuse will occur, and also the number of messages will also increase. "Good" block sizes are almost always in the [32 .. 256] interval and it depends on Cache size. These block size are found to be good, 80-216 for IA32; 128-192 for IA64 3M cache; 400 for 4M cache for IA64 and 130 for Woodcrests.

Process Grid Ratio (PXQ): This depends on physical interconnection network. P and Q should be approximately equal, with Q slightly larger than P. For e.g. for a 480 processor cluster, 20X24 will be a good ratio.

Tips: You can also try changing the node-order in the machine file for check the performance improvement. Choose all the above parameters by trial and error to get the best performance.

The following chart shows the linpack run performance results for various problem sizes for Intel® Xeon processor bases 5300 series (Clovertown) systems.

If you are building haply rather than using the binary from Intel Optimized linpack, make sure that, your MPI is running properly, FORTRAN, C++, MPI and MKL libraries are in LD_LIBRARY_PATH and FORTRAN, C++ and MPI binaries are in PATH.

• Intel® Cluster Tools
• Intel Xeon Processor- and Itanium 2-based Servers Homepage

Red Hat* Linux, Red Hat* Desktop Linux* 3, Red Hat* Enterprise Linux Desktop 4, Red Hat* Desktop 3 Update 4, Red Hat* Enterprise Linux Desktop 3 Update 3, Red Hat* Enterprise Linux Desktop 3 Update 4, Red Hat* Enterprise Linux Desktop 3 Update 5, Red Hat* Enterprise Linux Desktop 4 Update 1, Red Hat* Enterprise Linux 2.1, SUSE* Linux 9.1, SUSE* Linux Enterprise Server 8.0, SUSE* Linux Enterprise Server 9.0, Red Hat* Enterprise Linux 4.0, Redhat* Desktop 3 Update 5, Redhat* Desktop 3 Update 6, Redhat* Desktop 3 Update 7, Redhat* Desktop 4 Update 2, Redhat* Desktop 4 Update 3, Redhat* Desktop 4 Update 4, SuSE* Linux* Enterprise* Desktop 10, SUSE* Linux Enterprise Server 10, Red Hat* Linux 6.2, Red Hat* Linux 6.2 SBE2, Red Hat* Linux 7.0, Red Hat* Linux 7.1, Red Hat* Linux 7.2, Red Hat* Linux 7.3, SUSE* Linux 7.3, SUSE* Linux 8.0, SUSE* Linux 8.1, Red Hat* Linux 8.0, SUSE* Linux 7.2, SUSE* Linux 7.1, SUSE* Linux 7.0, SUSE* Linux, Red Hat* Linux Advanced Server 2.x, Red Hat* Linux 9.0, Red Hat* Enterprise Linux 3.0, SUSE* Linux* 8.2, Red Hat* Linux Advanced Server 3.x, SUSE* Linux* 9.x

This document provides the best known method for dealing with long diagnostic lists in Intel® Thread Checker.

Tips for Analyzing Long Diagnostic Lists [BKM.PDF]
Size: 2586113 bytes
Date: 03/14/03 02:38 PM

Note: PDF Files require Adobe Acrobat* Reader^†

^† This link will take you off of the Intel Web site. Intel does not control the content of the destination Web Site.

This document introduces threading concepts and provides a methodology for threading serial applications and for tuning threaded applications for performance using the Intel® Threading Tools.

	Threading Methodology: Practices & Principles [PDF]
	File Name: methodology.pdf Size: 1,684,645 bytes Date: 10/30/04

Note: PDF Files require Adobe Acrobat* Reader^†

^† This link will take you off of the Intel Web site. Intel does not control the content of the destination Web Site.