Distributed Memory Coarray Fortran with the Intel Fortran Compiler for Linux, versions 12.0-13.1: Essential Guide


THIS GUIDE IS OUT OF DATE: the guide for Intel Fortran v14.0 and above is HERE


THIS GUIDE APPLIES only to Composer XE 2011 (compiler version 12.0) through Composer XE 2013 (compiler version 13.1).  


Introduction :
This is an essential guide to using the Coarray Fortran (CAF) feature of the Intel Fortran Composer XE 2011 for Linux on a distributed memory cluster.

Version :
To use the distributed memory feature of Intel CAF, you must have a licensed copy of the Intel® Cluster Studio 2011 (Formerly Intel® Cluster Toolkit Compiler Edition).  The shared memory CAF feature is available in the Intel® Cluster Studio 2011, the Intel® Composer XE 2011 for Linux, and Intel® Visual Fortran Composer XE 2011 for Windows.  The CAF feature is currently not available under Mac OS X, and the distributed memory CAF feature is only available under the Linux operating system.
Requires Intel MPI for Linux version 4.0 Update 1 or greater
Requires Intel Composer XE 2011 (original or any Update)
Requires Intel Cluster Studio 2011 license for COMPILATION ONLY (no runtime checks).

Configuration Set Up :
In order to run a distributed memory Coarray Fortran application, you must have an established cluster and an installation of Intel MPI version or greater.  BEFORE ATTEMPTING TO RUN INTEL CAF, make sure you can run a simple MPI 'hello world' program on your cluster across multiple nodes using Intel MPI.

Successful configuration and running of MPI jobs under Intel MPI is a prerequisite to using the Intel Fortran CAF feature in distributed memory mode.  In order to get support on the Intel CAF feature you will be asked to demonstrate the ability to run a 'hello world' MPI program on your cluster.  Please read the Intel MPI Release Notes and Getting_Started documents that come with your Intel MPI installation in the <install directory>/doc/ directory.

Before running an Intel CAF application, perform the following steps:

1) set up a mpd.hosts fle:  If your cluster hosts are fixed and you do NOT run under a batch system with PBS, set up a static hosts file.  In your home directory, create a file with the names of your cluster hostnames, one host per line optionally with the number of cores (processors) on each host similar to this:

If you run under a batch system such as PBS you will not know the hosts in your allocation until job dispatch.  In this scenario we will use the --file option to mpdboot that will use a batch system supplied hosts file (see later in this document).

2) source Intel MPI and Intel Fortran compiler "...vars.sh" or "...vars.csh" files:  You must set up the paths to Intel MPI and Intel Fortran in your environment.  Furthermore, these should be sourced by child processes.  Thus it is recommended to perform the following source commands and/or add these to your .bashrc or .cshrc files in your home directory:

source <path to Intel MPI installation>/[ia32 | intel64]/bin/mpivars.sh
source <path to Intel Fortran installation>/bin/compilervars.sh [ia32 | intel64]
where you choose between 32 and 64 bit environments with ia32 or intel64 respectively.

3) Setup a Coarray Fortran (CAF) configuration file:  When you run a distributed memory Coarray Fortran program, the application first runs a job launcher.  The job launcher invoked to start a distributed memory CAF application will use the Intel MPI 'mpiexec' command to start the job on the hosts in the cluster.  This CAF job launcher will first read the "CAF configuration file" to pick up arguments that will be passed to the Intel MPI mpiexec command.  Thus, the "CAF configuration file" is nothing more than arguments to the Intel MPI 'mpiexec' command.  And example CAF configuration file may contain:
-envall -n 16 ./mycafprogram.exe

Where you want 16 CAF images created of program "mycafprogram.exe".  Read the Intel MPI documentation on mpiexec for all possible options.  Some common options:

-envall   copies your current environment variables to the environment of your CAF processes.  HIGHLY RECOMMENDED.

-n N     create N images (processes).  REQUIRED

-rr       round-robin image distribution of images to nodes.  OPTIONAL. Default is to pack each node with the number of images equal to the number of cores (real cores PLUS any hyperthreaded virtual cores) one host at at time in the order specified in mpd.hosts file.  Round-robin is one way to avoid using hyperthreaded cores.  With the -rr option to mpiexec, image 1 is assigned to host1 from your mpd.hosts or PBS_NODEFILE, image 2 to host2, etc. to image N on hostN, at which point the allocation cycles back to image N+1 on host1 and so on.

-perhost N    distribute images to hosts in groups of N.  OPTIONAL. This is another way to avoid hyperthreaded cores:  set N to the number of real cores on each host.  image 1..N are allocated on host1, images N+1 to N+N on host2, etc.

Building the Application :
You are now ready to compile your Coarray Fortran application.  Create or use an existing Coarray Fortran application.  A sample Coarray Fortran 'hello world' application is included in the <compiler install dir>Samples/en_US/Fortran/coarray_samples/ directory.

The essential compiler arguments to use for distributed memory coarray applications are:

ifort -coarray=distributed -corray-config-file=<CAF config filename>

Some essential notes:  -coarray=distributed is necessary to create a distibuted memory CAF application.  This option is only available on systems with a valid Intel Cluster Studio license.  Without this license you cannot create distributed memory Coarray Fortran applications - you can, however, create and use shared memory CAF applications with any existing Intel Composer XE 2011 for Linux or Windows license.

-coarray-config-file=<CAF configuration file>   this option is used to set tell the CAF job launcher where to find the configuration file to find runtime arguments to 'mpiexec'.  This file need not exist at the time of compilation.  This file is ONLY read at job launch.  Thus, it can be changed or modified between job runs to change the number of images along with any other valid control option to 'mpiexec'.  This give the programmer a way to change the number of images and other parameters without having to recompile the application.  A reasonable name for the file may be ~/cafconfig.txt, but the name of the file and location is up to the user to decide.  One essential note:  the executable name is hard-coded in the CAF config file, so be sure that the executable name in the config file matches the name you used with the 'ifort -o <name>' option.  Also, be sure to use either the full pathname to the executable OR the current directory "dot" name, such as './a.out' or './mycafprogram.exe' as examples.

Note:  -coarray-num-images=N  compiler option is ignored for -coarray=distributed.  This option is only used by shared memory Coarray Fortran applications.  The number of images for distributed memory CAF applications is ONLY controlled at job launch by the '-n N' option in the CAF config file.

Of course, you can include any other compiler options including all optimization options.

Running the Application :
Running an Intel CAF application involves 3 commands:
  1. mpdboot
  2. <running the application executable, for example ./a.out or ./mycafprogram.exe>
  3. mpdallexit

1) mpdboot: This command sets up the underlying runtime daemons used to control the various processes in your CAF application.  'mpdboot' sets up the runtime on the hosts in either your mpd.hosts file and/or from a file specified in the --file= option.  The mpdboot option needs a '-n <number of nodes>' argument.  For example, if you wish to run across 4 nodes, and there are 4 hosts in your mpd.host file:

mpdboot -n 4

will start the runtime daemons needed on all 4 nodes in your cluster or batch allocation.

If you are using a static list of hosts in your cluster, you create a mpd.hosts file as documented in the Configuration Set Up section above and place that in your current directory or your home directory, running mpdboot thusly:

mpdboot --file=~/mpd.hosts -n 4

for example for 4 nodes.

If you run on under PBS, the PBS batch system will create a hosts file for you with the list of hosts in your current batch allocation and set environment variable PBS_NODEFILE to point to this hosts file.  Thus, you can use this mpdboot command:

mpdboot --file=$PBS_NODEFILE -n 4

Other batch systems will similarly create a hosts file and set an appropriate environment variable.  Please consult your site system administrator for the environment variable and hosts file, as these host file conventions vary widely by batch system and local configuration.

Another mpdboot option often needed is '--rsh=[rsh | ssh]' where the remote connection method is specified.  Again, consult your site system administrator to determine if MPI uses RSH or SSH for remote connections.  In most cases ssh is used.   Thus, an example mpdboot command would look like:

mpdboot --rsh=ssh --file=~/mpd.hosts -n N
mpdboot --rsh=rsh --file=$PBS_NODEFILE -n N

CHECK:  run the command 'mpdtrace' or 'mpdtrace -l' to confirm that your mpdboot launched runtime daemons on each node and that each node is communicating.  If all is well, you will get a list of nodes participating in your runtime environment.  If not, you will get an error message such as:
mpdtrace: cannot connect to local mpd (/tmp/mpd2.console_rwgreen); possible causes:
1. no mpd is running on this host
2. an mpd is running but was started without a "console" (-n option)


2) Run your compiled Intel CAF application

simply invoke your compiled Intel CAF application.  This application contains a job launcher that invokes the Intel MPI 'mpiexec' command using arguments from your CAF config file.


along with any command line arguments to the application.

Need to change the number of images launched or the arguments to mpiexec?  Simply change the CAF config file settings in the text file.  Remember, the -coarray-config-file=  options used at compile time set the name an location for this text file.  You should use a name and location you can remember for this file, such as -coarray-config-file=~/cafconfig.txt
Then just add mpiexec options to ~/cafconfig.txt, for example
-perhost 2 -envall -n 64 ./a.out

Note:  environment variable FORT_COARRAY_NUM_IMAGES has not effect on distributed memory CAF applications.  This environment variable is only honored by a shared memory CAF image.  Only the -n option in the CAF config file is used to control the number of CAF images for a distributed memory CAF application.

Again, read the mpiexec documentation in the Intel MPI documentation set.

3) Cleanup/shutdown:  run command 'mpdallexit' to kill the cluster runtime daemons.  This is the opposite of the 'mpdboot' command.

Known Issues or Limitations :

Many clusters have multiple MPI implementations installed along with Intel MPI.  The PATH variable, LD_LIBRARY_PATH MUST have Intel MPI paths BEFORE any other MPI installed on your system.  Make sure to ONLY source the mpivars.sh script to set this correctly OR insure that the correct Intel MPI paths appear before other MPI paths.

Batch system notes:  In the above notes, we added the option '-envall' to the CAF config file.  This is an attempt to get your current working environment variables to be inherited by your spawned remote CAF processes.  This was done to help insure that your PATH and LD_LIBRARY_PATH contain the paths to Intel MPI and Intel Fortran AND those paths appear before other MPI and compilers on your system.  HOWEVER, some batch scheduling systems will not allow environment inheritence: in other words they will throw out your current environment variables and use defaults for these.  That is why we suggested adding the 'source <path to intel MPI>/[ia32 | intel64]/bin/mpivars.sh' to your .bashrc, .cshrc, or .bash_profile.  These dot files are invoked by each child process created, and hence, SHOULD set the PATH and LD_LIBRARY_PATH appropriately.   When in doubt, execute 'which mpiexec' interactively, or put 'echo `which mpiexec`' in your batch script to insure the Intel MPI mpiexec is being used.  Other MPI implementation 'mpiexec' commands cannot be used and will cause errors.

It is critical to insure that you can execute an Intel MPI application PRIOR to attempting to run an Intel CAF program.  Keep a simple MPI 'hello world' handy to debug your environment.  Here is a sample:
program hello_mpi
implicit none
include 'mpif.h'
integer :: size, rank, ierr, len
integer :: status
character*(MPI_MAX_PROCESSOR_NAME) name

call mpi_init(ierr)
call mpi_comm_size(MPI_COMM_WORLD, size, ierr)
call mpi_comm_rank(MPI_COMM_WORLD, rank, ierr)
call MPI_GET_PROCESSOR_NAME(name, len, ierr)

write(6, "(*(a,i3))") " MPI: size = ", size, " rank = ", rank
write(6, * ) "host is ", trim(name)

call mpi_finalize(ierr)

end program hello_mpi

Compile with:   mpiifort -o hello_mpi hello_mpi.f90
run with:   mpdboot --rsh=ssh --file=~/mpd.hosts -n 2 ; mpiexec -n 4 ./hello_mpi ; mpdallexit

and different number of processes for the -n argument.  Make sure you can run across all the nodes you believe are in your cluster or your batch allocation.

READ: the Intel MPI Release Notes and the Getting_Started.pdf documents that come with Intel MPI in the <installdir>/doc/ directory.


Our User Forums are great places to see current issues and to post questions:

Intel MPI User Forum
Intel Fortran User Forum


Para obtener información más completa sobre las optimizaciones del compilador, consulte nuestro Aviso de optimización.