Building VASP* with Intel® MKL, Intel® Compilers and Intel® MPI

Building VASP* with Intel® MKL , Intel® Compilers and Intel® MPI

Step 1 – Overview

This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI on Linux platforms.

VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudo potentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate.  [Ref: VASP]

Version Information

This application note was created to help users who benchmark clusters using VASP to also incorporate the latest version of Intel® MKL on Linux platforms on Xeon systems.  This application note is verified with VASP 5.3.5 and Intel® Compilers 15.0, Intel® MKL 11.2 and Intel® MPI 5.0.3

More information on VASP can be found from http://cms.mpi.univie.ac.at/vasp/

Step 2 – Downloading VASP Source Code

VASP is not public-domain or share-ware, and will be distributed only after a license contract has been signed. Please visit VASP homepage to know more details on obtaining the license.

Prerequisites:

  Intel C++ and Fortran Compilers, Intel MKL, Intel MPI

The above products are bundled in Intel® Parallel Studio XE 2015 Cluster Edition and is available for 30 days evaluation or for purchase at

http://www.intel.com/software/products/ 

Step 3 - Configuration

Use the following commands to extract the VASPfiles:

$tar –xvzf vasp.tgz vasp.lib.tgz

This will create  vasp and vasp.lib directories

Set the Intel software tools environment variables by running the following command assuming the default path installation and building for Intel64 platform:

$source /opt/intel/parallel_studio_xe_2015/bin/psxevars.sh intel64

Note:  This application note is written specifically for use with the Intel compilers and MPI.

Step 4 – Building VASP

a. Build libdmy.a

Change directory to vasp.x.x.lib

Modify the makefile.linux_ifc_P4 Makefile to point to the correct Intel FORTRAN compiler.

FC=ifort

Run the following command from vasp.x.x.lib, using the Makefile for linux using the Intel compiler.

$make –f makefile.linux_ifc_P4

After a successful compilation, libdmy.a will be built in the same directory.

b. Build VASP

Change directory to vasp.x.x

Edit the makefile.linux_ifc_P4 to link with Intel® MKL libraries and change the Fotran compiler

Under FORTRAN and C++ compiler and linker part of the makefile

FC= mpiifort

CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxIFC\" -DIFC \
     -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \
     -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK  –DMKL_ILP64

Change the FORTRAN flags section as shown here.

FFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw
........
...
OFLAG=-O3 -xCORE-AVX2
.......

The -xCORE-AVX2 is for enabling vectorization for a Haswell Architecture.  Please refer the Intel Compiler reference manual for architecture specific compiler flags, or you may use -xhost to enable the highest available SIMD instruction if you are building and running the VASP on the same platform. 

Point to use Intel MKL libraries by modifying the MKL section as shown below:   -mkl=cluster is an Intel compiler flag that to include Intel MKL libraries, that will link with Intel MKL BLAS, LAPACK, FFT, ScaLAPACK functions that are used in VASP.

MKLROOT=/opt/intel/composer_xe_2015/mkl
MKL_PATH=$(MKLROOT)/lib/intel64
....
.....

BLAS= -mkl=cluster

LAPACK= 
......
.....

Comment out the existing FFT3D line, FFTW wrappers are integrated with the Intel MKL 10.2 onwards, and you do not need to specify the wrapper libs in the FFT3D line, and will be taken care of by -mkl=cluster flag mentioned above.

#FFT3D = fftdfurth.o fftdlib.o

FFT3D= fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o 
INCS = -I$(MKLROOT)/include/fftw

Since the -mkl=cluster, includes MKL ScaLAPACK libraries also, it is not required to mentioned the ScaLAPACK libs.  Leave it as it is. 

SCA=

Run the following command to build VASP.

$make –f makefile.linux_ifc_P4

This will create the VASP executable in the current directory.

Step 5 - Running VASP

Run vasp by executing mpiexec command with your required parameters.  Below, for e.g., to run 48 processes use as shown with your workloads, and hostnames are mentioned in the machinefile.

$mpiexec.hydra –np 48 -f machinefile ./vasp

Appendix A -How to check whether vasp is linked with Intel MKL?

To confirm the successful linking of MKL with VASP, please run ldd on vasp as below.  Version numbers have been modified with "x" to allow generic information.

[vasp.5.3]$ ldd vasp
        linux-vdso.so.1 =>  (0x00007fff3e7f0000)
        libmkl_intel_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002ad1af89e000)
        libmkl_cdft_core.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_cdft_core.so (0x00002ad1b01b5000)
        libmkl_scalapack_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_scalapack_lp64.so (0x00002ad1b03dd000)
        libmkl_blacs_intelmpi_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so (0x00002ad1b0cc7000)
        libmkl_sequential.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_sequential.so (0x00002ad1b0efb000)
        libmkl_core.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_core.so (0x00002ad1b17db000)
        libiomp5.so => /opt/intel/composer_xe_201x.x.xxx/compiler/lib/intel64/libiomp5.so (0x00002ad1b3348000)
        libmpifort.so.12 => /opt/intel/impi/5.x.x.xxx/intel64/lib/libmpifort.so.12 (0x00002ad1b3685000)
        libmpi.so.12 => /opt/intel/impi/5.x.x.xxx/intel64/lib/libmpi.so.12 (0x00002ad1b390f000)
.......
....

Appendix B - Known Issues and Limitations

There could be a compilation error on nonlr.F file as reported below.

mpif90 -fc=ifort  -FR -names lowercase -assume byterecl  -O2 -ip   -I/opt/intel/compiler/2013_sp1.1.106/composer_xe_2013_sp1.1.106/mkl/include/fftw  -c nonlr.f90

nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type.   [LM]
             DO LM=1,LMMAXC
nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type.   [LM]
             DO LM=1,LMMAXC
----------------^
nonlr.F(3069): error #6063: An INTEGER or REAL data type is required in this context.   [LM]
             DO LM=1,LMMAXC
----------------^
nonlr.F(3206): error #6404: This name does not have a type, and must have an explicit type.   [LM]
             DO LM=1,LMMAXC
----------------^
nonlr.F(3206): error #6063: An INTEGER or REAL data type is required in this context.   [LM]
             DO LM=1,LMMAXC
----------------^

By applying the patch mentioned below to the nonlr.F, this compilation error can be fixed.

3002c3002
<     INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
---
>     INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND
3144c3144
<     INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND
---
>     INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND

Appendix C – References

VASP (Vienna Ab-Initio Package Simulation)

Intel® Math Kernel Library

Intel® Parallel Studio XE