Building VASP* with Intel® MKL , Intel® Compilers and Intel® MPI
Step 1 – Overview
This guide is intended to help users on how to build VASP (Vienna Ab-Initio Package Simulation) using Intel® Math Kernel Library (Intel® MKL) and Intel® Compilers and Intel® MPI on Linux platforms.
VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudo potentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate. [Ref: VASP]
This application note was created to help users who benchmark clusters using VASP to also incorporate the latest version of Intel® MKL on Linux platforms on Xeon systems. This application note is verified with VASP 5.3.5 and Intel® Compilers 15.0, Intel® MKL 11.2 and Intel® MPI 5.0.3
More information on VASP can be found from http://cms.mpi.univie.ac.at/vasp/
Step 2 – Downloading VASP Source Code
VASP is not public-domain or share-ware, and will be distributed only after a license contract has been signed. Please visit VASP homepage to know more details on obtaining the license.
Intel C++ and Fortran Compilers, Intel MKL, Intel MPI
Step 3 - Configuration
Use the following commands to extract the VASP files:
$tar –xvzf vasp.tgz vasp.lib.tgz
This will create vasp and vasp.lib directories
Set the Intel software tools environment variables by running the following command assuming the default path installation and building for Intel64 platform:
$source /opt/intel/parallel_studio_xe_2015/bin/psxevars.sh intel64
Note: This application note is written specifically for use with the Intel compilers and MPI.
Step 4 – Building VASP
a. Build libdmy.a
Change directory to vasp.x.x.lib
Modify the makefile.linux_ifc_P4 Makefile to point to the correct Intel FORTRAN compiler.
Run the following command from vasp.x.x.lib, using the Makefile for linux using the Intel compiler.
$make –f makefile.linux_ifc_P4
After a successful compilation, libdmy.a will be built in the same directory.
b. Build VASP
Change directory to vasp.x.x
Edit the makefile.linux_ifc_P4 to link with Intel® MKL libraries and change the Fotran compiler
Under FORTRAN and C++ compiler and linker part of the makefile
CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \ -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf \ -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK –DMKL_ILP64
Change the FORTRAN flags section as shown here.
FFLAGS = -FR -names lowercase -assume byterecl -I$(MKLROOT)/include/fftw ........ ... OFLAG=-O3 -xCORE-AVX2 .......
The -xCORE-AVX2 is for enabling vectorization for a Haswell Architecture. Please refer the Intel Compiler reference manual for architecture specific compiler flags, or you may use -xhost to enable the highest available SIMD instruction if you are building and running the VASP on the same platform.
Point to use Intel MKL libraries by modifying the MKL section as shown below: -mkl=cluster is an Intel compiler flag that to include Intel MKL libraries, that will link with Intel MKL BLAS, LAPACK, FFT, ScaLAPACK functions that are used in VASP.
MKLROOT=/opt/intel/composer_xe_2015/mkl MKL_PATH=$(MKLROOT)/lib/intel64 .... ..... BLAS= -mkl=cluster LAPACK= ...... .....
Comment out the existing FFT3D line, FFTW wrappers are integrated with the Intel MKL 10.2 onwards, and you do not need to specify the wrapper libs in the FFT3D line, and will be taken care of by -mkl=cluster flag mentioned above.
#FFT3D = fftdfurth.o fftdlib.o
FFT3D= fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o INCS = -I$(MKLROOT)/include/fftw
Since the -mkl=cluster, includes MKL ScaLAPACK libraries also, it is not required to mentioned the ScaLAPACK libs. Leave it as it is.
Run the following command to build VASP.
$make –f makefile.linux_ifc_P4
This will create the VASP executable in the current directory.
Step 5 - Running VASP
Run vasp by executing mpiexec command with your required parameters. Below, for e.g., to run 48 processes use as shown with your workloads, and hostnames are mentioned in the machinefile.
$mpiexec.hydra –np 48 -f machinefile ./vasp
Appendix A - How to check whether vasp is linked with Intel MKL?
To confirm the successful linking of MKL with VASP, please run ldd on vasp as below. Version numbers have been modified with "x" to allow generic information.
[vasp.5.3]$ ldd vasp linux-vdso.so.1 => (0x00007fff3e7f0000) libmkl_intel_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002ad1af89e000) libmkl_cdft_core.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_cdft_core.so (0x00002ad1b01b5000) libmkl_scalapack_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_scalapack_lp64.so (0x00002ad1b03dd000) libmkl_blacs_intelmpi_lp64.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so (0x00002ad1b0cc7000) libmkl_sequential.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_sequential.so (0x00002ad1b0efb000) libmkl_core.so => /opt/intel/composer_xe_201x.x.xxx/mkl/lib/intel64/libmkl_core.so (0x00002ad1b17db000) libiomp5.so => /opt/intel/composer_xe_201x.x.xxx/compiler/lib/intel64/libiomp5.so (0x00002ad1b3348000) libmpifort.so.12 => /opt/intel/impi/5.x.x.xxx/intel64/lib/libmpifort.so.12 (0x00002ad1b3685000) libmpi.so.12 => /opt/intel/impi/5.x.x.xxx/intel64/lib/libmpi.so.12 (0x00002ad1b390f000) ....... ....
Appendix B - Known Issues and Limitations
There could be a compilation error on nonlr.F file as reported below.
mpif90 -fc=ifort -FR -names lowercase -assume byterecl -O2 -ip -I/opt/intel/compiler/2013_sp1.1.106/composer_xe_2013_sp1.1.106/mkl/include/fftw -c nonlr.f90 nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type. [LM] DO LM=1,LMMAXC nonlr.F(3069): error #6404: This name does not have a type, and must have an explicit type. [LM] DO LM=1,LMMAXC ----------------^ nonlr.F(3069): error #6063: An INTEGER or REAL data type is required in this context. [LM] DO LM=1,LMMAXC ----------------^ nonlr.F(3206): error #6404: This name does not have a type, and must have an explicit type. [LM] DO LM=1,LMMAXC ----------------^ nonlr.F(3206): error #6063: An INTEGER or REAL data type is required in this context. [LM] DO LM=1,LMMAXC ----------------^
By applying the patch mentioned below to the nonlr.F, this compilation error can be fixed.
3002c3002 < INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND --- > INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND 3144c3144 < INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, LM, IND --- > INTEGER IP, LMBASE, ISPIRAL, ISPINOR, NLIIND, NIS, NT, LMMAXC, NI, INDMAX, L, IND
Appendix C – References
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804