Building WRF with the Intel® compilers

Version Information

This application note was created to help users of WRF version 2.2 and 2.2.1 make use of the Intel Fortran compiler, versions 10.1 and 11. These instructions have been tested on Intel® Core®2 Duo processors and Intel Itanium® 2 processors running Linux and on Intel® Core®2 Duo processors running Mac OS* X version 10.5.

For WRF 3.0 and later, please review the article Building WRF and WPS with the Intel® Compilers on Linux.


Application Notes

The Weather Research and Forecasting (WRF) Model is a next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. Parallel implementations of WRF include support for OpenMP* and for MPI*. For further information, see http://wrf-model.org/index.php


Obtaining the Source Code

WRF is in the public domain and source code may be obtained from the WRF project at the URL above. The version with the ARW solver, discussed here, may be downloaded from http://www.mmm.ucar.edu/wrf/users/downloads.html†, as well as test data.


Prerequisites

  • Intel® Fortran Compiler for Linux (version 10.1 or 11.0)
  • Intel® C++ Compiler for Linux (version 10.1 or 11.0)
  • An MPI library, such as Intel MPI, if intending to build a distributed memory version.
  • See the WRF website for a full list of prerequisite libraries.

Configuration and Set-up Information

  1. Set up the Intel® compiler environment, e.g., by the bash shell command source ifortvars.sh from the compiler bin directory. Also source iccvars.sh if using the Intel C++ compiler. For the 11.0 compiler only, these scripts require an argument intel64 or ia32.
  2. If using Intel® MPI, set up the environment with source mpivars.sh from the MPI bin directory (bin64 directory for Intel 64).
  3. Untar the WRF download and configure and build WRF according to the README file:
    • Run ./configure for WRF and select an option that includes ifort.
    • Modify the configure.wrf file as necessary. We recommend:
      • Replace " -mp" by " -fp-model precise"
      • For Intel Itanium-based processors running Linux, set FCOPTIM = -O3 –fno-alias -ip
      • For Intel® 64 or IA-32 processors running Linux or Mac OS X, set: FCOPTIM = -O3 –xT –fno-alias -ip for Intel® Core® 2 Duo processors, FCOPTIM = -O3 –xP –fno-alias -ip for any Intel® processor with at least SSE3 support. For the 11.0 compiler, -xssse3 is equivalent to -xT and -xsse3 is equivalent to -xP.
      • For Intel® 64 or IA-32 processors running Linux, set: FCOPTIM = -O3 –xW –fno-alias -ip for any processor with at least SSE2 support. For the 11.0 compiler, -msse2 is equivalent to –xW and is the default setting for Intel® 64 or IA-32 processors running Linux.
      • If ARCHFLAGS contains the definition –DIFORT_KLUDGE, remove it.
      • Ensure that the base options include –convert big-endian and –align all
      • Verify NETCDFPATH
      • Verify path for MPI if used.
      • Make any additional changes indicated in the "Known Issues" section.

With these options, it should be possible to build all source files with full optimization. However, if desired, certain files, such as module_dm, may be built with FCBASEOPTS and OMP but without FCOPTIM, in order to reduce compilation time and memory requirement.

The -O3 option is available for both Intel® and non-Intel microprocessors but it may result in more optimizations for Intel microprocessors than for non-Intel microprocessors. For more information on processor-specific optimizations, see Intel® compiler options for SSE generation and processor-specific optimizations.


Building WRF

Choose one of the WRF test examples, downloading any required data, and build the test, preserving the output, for example:

./compile em_real > build.log 2>&1

Running WRF

The following example runs the real data test, real.exe. The "real" data test requires data downloaded from the WRF web site, the "ideal" tests do not. Untar the real data files and run the initialization code to generate WRF input files. Be sure to increase the shell stack limit (bash shell).

cd test/em_real
tar –xzvf jan00_wps.tar.gz
./real.exe
ulimit –s unlimited

Run the main simulation (without Intel® MPI):

./wrf.exe

Run the main simulation (with Intel® MPI):

mpiexec –n <number of procs> ./wrf.exe

Verifying Correctness

See the README_TEST_CASES file for the ideal test cases. The utility <install dir>/external/io_netcdf/diffwrf may be used to compare an output file, such as <install dir>/test/em_real/wrfout_d01_2000-01-24_12:00:00, to a reference version. See the WRF website for further details.


Known Issues and Limitations

On IA-64: "fortcom: Warning: Optimization suppressed due to excessive resource requirements"
For certain WRF configurations, typically involving RSL, the compiler may scale back optimizations to limit the memory requirement and compile time. If you are building WRF on a system with plenty of memory, say 8 GB, you may use the option –override-limits to ask the compiler to continue the compilation without reducing the optimization level. When building with OpenMP, the file solve_em.f90 should be compiled with –override-limits, whatever the optimization level.

On Intel 64: "Fatal compilation error: Out of memory asking for ……."
The compiler for Intel 64 is a 32 bit executable and can access a maximum of 4 GB of memory. For certain WRF configurations, typically involving RSL, the compiler may exhaust the available memory for one or two files when compiled with maximum optimization. This may occur for additional files if less than 4GB total (physical + virtual) memory is available, or on IA-32. In version 10.1 of the compiler only, the internal switch –switch fe_use_rtl_copy_arg_inout may be used to reduce the memory requirement. If warning messages such as "An internal threshold was exceeded" are seen, the additional option –mP2OPT_vec_xform_level=103 may be used to preserve optimization levels.

The version 11.0 compiler for Intel 64 is a native 64 bit executable and is not subject to the 4GB limit on address space. The compiler may still exceed internal limits that are intended to limit memory usage and reduce compile time; this may or may not be accompanied by a warning message. These limits may be avoided by the switch -override-limits in the 11.0 and later compilers. It is strongly recommended to compile the file solve_em.f90 using the switch –override-limits. On a system with plenty of memory, -override-limits may be included in FCOPTIM

Large memory use or very long compile times for module_configure.f90 may indicate that the definition –DIFORT_KLUDGE has not been removed from the configuration file as described above.

For more complete information about compiler optimizations, see our Optimization Notice.

16 comments

Top
Marco B.'s picture

Perhaps the information contained in this discussion are no longer entirely up to date in the year 2014.

Someone could kindly review them in the light of the updates gained in Intel processors, Intel compilers, WRF model (v. 3.6) and libraries?

Many thanks in advance!

Marco

kiran s.'s picture

i got error while compiling wrfv3.5

zlib-1.2.8

./configure –prefix=/home1/cdac/dependency/support

Vi Makefile

Change gcc to mpiicc

Make

Make install

Hdf5:

./configure --prefix=/home1/cdac/dependancy/support/ CC=icc FC=ifort FC=-fPIC CFLAGS=-fPIC LIBS=-lz CPPFLAGS=-I/home1/cdac/dependancy/support/include/ LDFLAGS=-L/home1/cdac/dependancy/support/lib/ --enable-fortran=ifort CXX=icpc --enable-cxx

Make

Make install

NETCDF4:

netcdf-4.1.3

./configure --prefix=/home1/cdac/dependancy/support/ --enable-netcdf-4 --with-hdf5=/home1/cdac/dependancy/support/ --enable-separate-fortran CC=icc FC=ifort F77=ifort FCFLAGS=-fPIC CFLAGS=-fPIC CXX=icpc CXXFLAGS=-fPIC CPPFLAGS=-I/home1/cdac/dependancy/support/include/ LDFLAGS=-L/home1/cdac/dependancy/support/lib/ LIBS=-lz –fPIC

Make

Make install

 

NETCDF-FORTRAN:

./configure --prefix=/home1/cdac/dependancy/support/ CC=icc FC=ifort F77=ifort FFLAGS=-fPIC FCFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-I/home1/cdac/dependancy/support/include/ LDFLAGS=-L/home1/cdac/dependancy/support/lib/ LIBS=-lz –fPIC

Make

Make install

all are ok no error in make of above but error in wrfv3.5

cd wrfv3

./configure

select all intel choice 19 (dmpr)

vi configure.wrf

change icc to mpiicc and mpiifort

./compile em_real (gives error)

ar: module_bl_temf.o: No such file or directory
make[2]: [physics] Error 1 (ignored)
make[2]: Leaving directory `/home1/cdac/climate/wrf/WRFV3/phys'
make[1]: Leaving directory `/home1/cdac/climate/wrf/WRFV3'

2nd error

start_em.f90:(.text+0x218ab): undefined reference to `module_diagnostics_mp_pld_'
start_em.f90:(.text+0x21cfb): undefined reference to `module_fr_fire_driver_wrf_mp_fire_driver_em_init_'
make[1]: [em_wrf] Error 1 (ignored)
make[1]: Leaving directory `/home1/cdac/climate/wrf/WRFV3/main'
( cd run ; /bin/rm -f wrf.exe ; ln -s ../main/wrf.exe . )

 

3rd error

start_em.f90:(.text+0x218ab): undefined reference to `module_diagnostics_mp_pld_'
start_em.f90:(.text+0x21cfb): undefined reference to `module_fr_fire_driver_wrf_mp_fire_driver_em_init_'
make[1]: [em_real] Error 1 (ignored)
mpiifort -f90=mpiifort -o nup.exe -openmp -fpp -auto -O3 -ip -fp-model precise -w -ftz -align all -fno-alias -FR -convert big_endian

4 th error

module_wrf_error.f90:(.text+0x1b2a): undefined reference to `__kmpc_end_master'
module_wrf_error.f90:(.text+0x1b42): undefined reference to `__kmpc_master'
module_wrf_error.f90:(.text+0x1b9b): undefined reference to `__kmpc_end_master'
make[2]: [diffwrf] Error 1 (ignored)
make[2]: Leaving directory `/home1/cdac/climate/wrf/WRFV3/external/io_int'
make[1]: Leaving directory `/home1/cdac/climate/wrf/WRFV3'
make -i -r MODULE_DIRS="-I../dyn_em -I../dyn_nmm   -I/home1/cdac/climate/wrf/WRFV3/external/esmf_time_f90  -I/home1/cdac/climate/wrf/WRFV3/main -I/home1/cdac/climate/wrf/WRFV3/external/io_netcdf -I/home1/cdac/climate/wrf/WRFV3/external/io_i

5 th error

cdac/dependancy/support//include  -i4  module_bl_temf.f90
catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for module_bl_temf.f90 (code 1)
make[3]: [module_bl_temf.o] Error 1 (ignored)
rm -f module_bl_gbmpbl.o
/lib/cpp -C -P -I/home1/cdac/climate/wrf/WRFV3/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DCOAMPS_CORE=0 -DDA_CORE=0 -DEXP_CORE=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 

 

 

kiran s.'s picture

libtool: link: warning: `/home1/cdac/dependancy/support/lib//libhdf5_hl.la' seems to be moved
libtool: link: warning: `/home1/cdac/dependancy/support/lib//libhdf5.la' seems to be moved
libtool: link: ifort -shared -nofor_main  .libs/fort-attio.o .libs/fort-control.o .libs/fort-dim.o .libs/fort-genatt.o .libs/fort-geninq.o .libs/fort-genvar.o .libs/fort-lib.o .libs/fort-misc.o .libs/fort-v2compat.o .libs/fort-vario.o .libs/fort-var1io.o .libs/fort-varaio.o .libs/fort-varmio.o .libs/fort-varsio.o .libs/fort-nc4.o  -Wl,--whole-archive ../f90/.libs/libnetcdff90.a -Wl,--no-whole-archive  -Wl,-rpath -Wl,/home1/cdac/dependancy/netcdf-4.1.3/liblib/.libs -Wl,-rpath -Wl,/home1/cdac/dependancy/support/lib -Wl,-rpath -Wl,/home1/cdac/dependancy/support/lib -L/home1/cdac/dependancy/support/lib/ ../liblib/.libs/libnetcdf.so -L/home1/cdac/dependancy/support/lib -L/home1/cdac/dependancy/test/lib/ /home1/cdac/dependancy/test/lib/libhdf5_hl.a -L/home1/cdac/dependancy/test/lib//lib -L/home1/cdac/dependancy/test/lib /home1/cdac/dependancy/test/lib/libhdf5.a /home1/cdac/dependancy/support/lib/libhdf5_hl.so /home1/cdac/dependancy/support/lib/libhdf5.so -lrt -lz -lcurl -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64 -L/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64 -L/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/ -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/ -L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/ -L/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64/ -L/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64/ -L/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4/ -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib -lifport -lifcore -limf -lm -lipgo -lintlc -lpthread -lsvml -lgcc -lgcc_s -lirc_s -ldl -lc    -Wl,-soname -Wl,libnetcdff.so.5 -o .libs/libnetcdff.so.5.1.0
ld: /home1/cdac/dependancy/test/lib/libhdf5_hl.a(H5DS.o): relocation R_X86_64_32 against `.rodata.str1.4' can not be used when making a shared object; recompile with -fPIC
/home1/cdac/dependancy/test/lib/libhdf5_hl.a: could not read symbols: Bad value
make[3]: *** [libnetcdff.la] Error 1
make[3]: Leaving directory `/home1/cdac/dependancy/netcdf-4.1.3/fortran'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home1/cdac/dependancy/netcdf-4.1.3/fortran'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home1/cdac/dependancy/netcdf-4.1.3'
make: *** [all] Error 2

please help 

i have compiled zlib with icc and hdf icc ifort and flag -fPIC but while instaling netcdf it gives me above error

 

hari k.'s picture

Dear sir,
 greetings of the hour.
  i am getting the same error what i need to edit in profile file are what i can do to resolve this issue.present i am using inlet compiler ICC i fort and my process is ia64. and m no of processor 8.
 Waiting for our reply.

Thank you.

 

 

Error: A license for FCompL could not be obtained.

Your license has expired.

License file(s) used were (in this order):
    1.  /opt/intel/cc/10.0.023/licenses/*.lic
    2.  /opt/intel/licenses/EVAL_L_CMP_CPP_VZ63-G66JJJ68.lic
    3.  /opt/intel/licenses/EVAL_L_CMP_FOR_V6XF-WZSGJRLM.lic
    4.  /opt/intel/licenses/EVAL_L__CPP_V3N3-GXP8NTJZ.lic
    5.  /opt/intel/licenses/EVAL_L__FOR_VTVV-S36JGVR8.lic
    6.  /opt/intel/licenses/ifor1.lic
    7.  /root/intel/licenses
    8.  /opt/intel/fc/10.0.023/licenses/*.lic
    9.  /opt/intel/fc/10.0.023/bin/*.lic

Please visit http://www.intel.com/software/products to obtain license renewal information.

ifort: error #10052: could not checkout FLEXlm license
make[2]: [module_io_int_read.o] Error 1 (ignored)
rm -f libwrfio_int.a
INTERNAL_BUILD_ERROR_SHOULD_NOT_NEED_AR rv libwrfio_int.a io_int.o io_int_idx.o module_io_int_idx.o module_io_int_read.o
make[2]: INTERNAL_BUILD_ERROR_SHOULD_NOT_NEED_AR: Command not found
make[2]: [libwrfio_int.a] Error 127 (ignored)
ranlib libwrfio_int.a
ranlib: 'libwrfio_int.a': No such file
if [ -f ../../frame/pack_utils.o -a -f ../../frame/clog.o ] ; then \
  ifort -ip -fp-model precise -w -ftz -align all -fno-alias -FR -convert big_endian     -I../ioapi_share -o diffwrf diffwrf.f \
        ../../frame/pack_utils.o ../../frame/module_internal_header_util.o ../../frame/module_driver_constants.o ../../frame/module_machine.o ../../frame/module_wrf_error.o ../../frame/wrf_debug.o -L/data/CWRF/WRFV3/external/esmf_time_f90 -lesmf_time libwrfio_int.a ; \
fi
make[2]: Leaving directory `/data/CWRF/WRFV3/external/io_int'
make[1]: Leaving directory `/data/CWRF/WRFV3'
make -i -r MODULE_DIRS="-I../dyn_em -I../dyn_nmm   -I/data/CWRF/WRFV3/external/esmf_time_f90  -I/data/CWRF/WRFV3/main -I/data/CWRF/WRFV3/external/io_netcdf -I/data/CWRF/WRFV3/external/io_int -I/data/CWRF/WRFV3/frame -I/data/CWRF/WRFV3/share -I/data/CWRF/WRFV3/phys -I/data/CWRF/WRFV3/chem -I/data/CWRF/WRFV3/inc -I/include " shared
make[1]: Entering directory `/data/CWRF/WRFV3'
--------------------------------------
if [ "`echo -j 2 | sed -e 's/-j//g' -e 's/ \+//g'`" -gt "6" ] ; then \
  ( cd share ; make -i -r -j 6 ) ;  \
else \
  ( cd share ; make -i -r -j 2 ) ;  \
fi
make[2]: Entering directory `/data/CWRF/WRFV3/share'
rm -f module_model_constants.o
make[2]: *** No rule to make target `/data/CWRF/WRFV3/external/esmf_time_f90/module_utility.o', needed by `module_bc_time_utilities.o'.  Stop.
make[2]: *** Waiting for unfinished jobs....
/lib/cpp -C -P -I/data/CWRF/WRFV3/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DCOAMPS_CORE=0 -DDA_CORE=0 -DEXP_CORE=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_FUNC -DWRF_USE_CLM -DNO_IEEE_MODULE  -DDM_PARALLEL -DUSE_ALLOCATABLES -DGRIB1 -DINTIO -DLIMIT_ARGS -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0  -I. -traditional   module_model_constants.F  > module_model_constants.bb
make[2]: *** Waiting for unfinished jobs....
/data/CWRF/WRFV3/tools/standard.exe module_model_constants.bb | /lib/cpp -C -P > module_model_constants.f90
make[2]: *** Waiting for unfinished jobs....
/bin/sh: /data/CWRF/WRFV3/tools/standard.exe: No such file or directory
rm -f module_model_constants.b module_model_constants.bb
make[2]: *** Waiting for unfinished jobs....
make[2]: *** Waiting for unfinished jobs....
mpif90  -o module_model_constants.o -c -O3 -ip -fp-model precise -w -ftz -align all -fno-alias -FR -convert big_endian    -I../dyn_em -I../dyn_nmm   -I/data/CWRF/WRFV3/external/esmf_time_f90  -I/data/CWRF/WRFV3/main -I/data/CWRF/WRFV3/external/io_netcdf -I/data/CWRF/WRFV3/external/io_int -I/data/CWRF/WRFV3/frame -I/data/CWRF/WRFV3/share -I/data/CWRF/WRFV3/phys -I/data/CWRF/WRFV3/chem -I/data/CWRF/WRFV3/inc -I/include  -i4  module_model_constants.f90
make[2]: *** Waiting for unfinished jobs....
gfortran: precise: No such file or directory
gfortran: all: No such file or directory
gfortran: big_endian: No such file or directory
gfortran: unrecognized option '-convert'
f951: error: unrecognized command line option "-align"
f951: error: unrecognized command line option "-fp-model"
f951: error: unrecognized command line option "-ftz"
f951: error: unrecognized command line option "-fno-alias"
make[2]: [module_model_constants.o] Error 1 (ignored)
make[2]: Leaving directory `/data/CWRF/WRFV3/share'
make[1]: [shared] Error 2 (ignored)
make[1]: Leaving directory `/data/CWRF/WRFV3'
make -i -r MODULE_DIRS="-I../dyn_em -I../dyn_nmm   -I/data/CWRF/WRFV3/external/esmf_time_f90  -I/data/CWRF/WRFV3/main -I/data/CWRF/WRFV3/external/io_netcdf -I/data/CWRF/WRFV3/external/io_int -I/data/CWRF/WRFV3/frame -I/data/CWRF/WRFV3/share -I/data/CWRF/WRFV3/phys -I/data/CWRF/WRFV3/chem -I/data/CWRF/WRFV3/inc -I/include " physics
make[1]: Entering directory `/data/CWRF/WRFV3'

 

Scott C.'s picture

I am running WRF v3.4.1 on the following shared memory system:
Linux Ubuntu 12.04
32 x 2.2GHz Xeon E5-4600 Series CPU Cores / 64MB Intel Smart Cache
128GB DDR3 1600Mhz ECC REG System Memory

I have installed the ComposerXE fortran and C/C++ compilers and am compiling WRF using the smpar option.

It runs fine using OMP_NUM_THREADS=1. However, with any other OMP_NUM_THREADS value, I get the following error message:
call rk_phys_bc_dry_1
call rk_phys_bc_dry_1
call init_zero_tendency
forrtl: severe (40): recursive I/O operation, unit -1, file unknown

Any help is appreciated.

Scott

Mikhail Shiryaev's picture

Sorry but I don't use another linux distributives.
This solution (-heap-arrays) i found on http://forum.wrfforum.com/viewtopic.php?f=6&t=1625&start=0

salmontres's picture

Hi Mikhail,

Do you know if you have to use the -heap-arrays option on CentOS as well? I'm getting seg faults after successfully compiling WRF 3.3 for smpar and dmpar, and I have no idea what could be the cause.

Mikhail Shiryaev's picture

To compile on Ubuntu (10.04_x64) need add to CFLAGS_LOCAL and FCOPTIM "-heap-arrays" to avoid the segmentation fault.
Sorry about my not good english.

anonymous's picture

You completed several good points there.
I did a search on the theme and found most persons will consent with your blog.

anonymous's picture

we have been running wrf on amd (opteron) processors with pgroup compiler for a long period. Now we are trying to move to intel. So we are using the latest version of intel, we compiuled mpich2 also with ifort, and wrf has been compiled successfully but the model crashes just after writting the initial wrfout and before being able to give any timestep. Is there a known problem with amd (opteron or phanom) with wrf/intel execution? For instance we were able to run mm5 (parallel) compiled with intel in the same processors without any problem. Any ideas?

thank you in advance
V. Kotroni
National Observatory of Athens
Greece

Pages

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.