Problem : The first time the compiler is run after a login or after not being run for several minutes, this initial compilation can take dramatically longer than subsequent compilations. Subsequent compilations are significantly faster

Environment : RedHat Enterprise Linux and its derivativatives


Root Cause : Full look up of multiple directories causes timeout.


Resolution1 : Remove as many files and directory as you can from /tmp 
The slowness of the first compilation is due to the license manager examining every file on /tmp. This can initially take several seconds as this information is not iniitally cached by the OS. To avoid long delays, remove all unnecessary files from /tmp to speed up this process. Or see Resolution 2 below to improve the speed of the 'stat' operation on /tmp.
 

Resolution2 : Modify you $LS_OPTIONS environment variable to --color=none -U
This is one of the faster ls option settings. It will prevent you from grabbing all inode information unless you explicitly want it.

Introduction :
This is an essential guide to using the Coarray Fortran (CAF) feature of the Intel Fortran Composer XE 2011 for Linux on a distributed memory cluster.

Version :
To use the distributed memory feature of Intel CAF, you must have a licensed copy of the Intel® Cluster Studio 2011 (Formerly Intel® Cluster Toolkit Compiler Edition).  The shared memory CAF feature is available in the Intel® Cluster Studio 2011, the Intel® Composer XE 2011 for Linux, and Intel® Visual Fortran Composer XE 2011 for Windows.  The CAF feature is currently not available under Mac OS X, and the distributed memory CAF feature is only available under the Linux operating system.
Requires Intel MPI for Linux version 4.0 Update 1 or greater
Requires Intel Composer XE 2011 (original or any Update)
Requires Intel Cluster Studio 2011 license for COMPILATION ONLY (no runtime checks).

Configuration Set Up :
In order to run a distributed memory Coarray Fortran application, you must have an established cluster and an installation of Intel MPI version 4.0.1.007 or greater.  BEFORE ATTEMPTING TO RUN INTEL CAF, make sure you can run a simple MPI 'hello world' program on your cluster across multiple nodes using Intel MPI.

Successful configuration and running of MPI jobs under Intel MPI is a prerequisite to using the Intel Fortran CAF feature in distributed memory mode.  In order to get support on the Intel CAF feature you will be asked to demonstrate the ability to run a 'hello world' MPI program on your cluster.  Please read the Intel MPI Release Notes and Getting_Started documents that come with your Intel MPI installation in the <install directory>/doc/ directory.

Before running an Intel CAF application, perform the following steps:

1) set up a mpd.hosts fle:  If your cluster hosts are fixed and you do NOT run under a batch system with PBS, set up a static hosts file.  In your home directory, create a file with the names of your cluster hostnames, one host per line optionally with the number of cores (processors) on each host similar to this:
nodehostname1
nodehostname2
...
nodehostnameN

If you run under a batch system such as PBS you will not know the hosts in your allocation until job dispatch.  In this scenario we will use the --file option to mpdboot that will use a batch system supplied hosts file (see later in this document).

2) source Intel MPI and Intel Fortran compiler "...vars.sh" or "...vars.csh" files:  You must set up the paths to Intel MPI and Intel Fortran in your environment.  Furthermore, these should be sourced by child processes.  Thus it is recommended to perform the following source commands and/or add these to your .bashrc or .cshrc files in your home directory:

source <path to Intel MPI installation>/[ia32 | intel64]/bin/mpivars.sh
source <path to Intel Fortran installation>/bin/compilervars.sh [ia32 | intel64]
where you choose between 32 and 64 bit environments with ia32 or intel64 respectively.

3) Setup a Coarray Fortran (CAF) configuration file:  When you run a distributed memory Coarray Fortran program, the application first runs a job launcher.  The job launcher invoked to start a distributed memory CAF application will use the Intel MPI 'mpiexec' command to start the job on the hosts in the cluster.  This CAF job launcher will first read the "CAF configuration file" to pick up arguments that will be passed to the Intel MPI mpiexec command.  Thus, the "CAF configuration file" is nothing more than arguments to the Intel MPI 'mpiexec' command.  And example CAF configuration file may contain:
-envall -n 16 ./mycafprogram.exe

Where you want 16 CAF images created of program "mycafprogram.exe".  Read the Intel MPI documentation on mpiexec for all possible options.  Some common options:

-envall   copies your current environment variables to the environment of your CAF processes.  HIGHLY RECOMMENDED.

-n N     create N images (processes).  REQUIRED

-rr       round-robin image distribution of images to nodes.  OPTIONAL. Default is to pack each node with the number of images equal to the number of cores (real cores PLUS any hyperthreaded virtual cores) one host at at time in the order specified in mpd.hosts file.  Round-robin is one way to avoid using hyperthreaded cores.  With the -rr option to mpiexec, image 1 is assigned to host1 from your mpd.hosts or PBS_NODEFILE, image 2 to host2, etc. to image N on hostN, at which point the allocation cycles back to image N+1 on host1 and so on.

-perhost N    distribute images to hosts in groups of N.  OPTIONAL. This is another way to avoid hyperthreaded cores:  set N to the number of real cores on each host.  image 1..N are allocated on host1, images N+1 to N+N on host2, etc.


Building the Application :
You are now ready to compile your Coarray Fortran application.  Create or use an existing Coarray Fortran application.  A sample Coarray Fortran 'hello world' application is included in the <compiler install dir>Samples/en_US/Fortran/coarray_samples/ directory.

The essential compiler arguments to use for distributed memory coarray applications are:

ifort -coarray=distributed -corray-config-file=<CAF config filename>

Some essential notes:  -coarray=distributed is necessary to create a distibuted memory CAF application.  This option is only available on systems with a valid Intel Cluster Studio license.  Without this license you cannot create distributed memory Coarray Fortran applications - you can, however, create and use shared memory CAF applications with any existing Intel Composer XE 2011 for Linux or Windows license.

-coarray-config-file=<CAF configuration file>   this option is used to set tell the CAF job launcher where to find the configuration file to find runtime arguments to 'mpiexec'.  This file need not exist at the time of compilation.  This file is ONLY read at job launch.  Thus, it can be changed or modified between job runs to change the number of images along with any other valid control option to 'mpiexec'.  This give the programmer a way to change the number of images and other parameters without having to recompile the application.  A reasonable name for the file may be ~/cafconfig.txt, but the name of the file and location is up to the user to decide.  One essential note:  the executable name is hard-coded in the CAF config file, so be sure that the executable name in the config file matches the name you used with the 'ifort -o <name>' option.  Also, be sure to use either the full pathname to the executable OR the current directory "dot" name, such as './a.out' or './mycafprogram.exe' as examples.

Note:  -coarray-num-images=N  compiler option is ignored for -coarray=distributed.  This option is only used by shared memory Coarray Fortran applications.  The number of images for distributed memory CAF applications is ONLY controlled at job launch by the '-n N' option in the CAF config file.

Of course, you can include any other compiler options including all optimization options.


Running the Application :
Running an Intel CAF application involves 3 commands:

1. mpdboot
2. <running the application executable, for example ./a.out or ./mycafprogram.exe>
3. mpdallexit

1) mpdboot: This command sets up the underlying runtime daemons used to control the various processes in your CAF application.  'mpdboot' sets up the runtime on the hosts in either your mpd.hosts file and/or from a file specified in the --file= option.  The mpdboot option needs a '-n <number of nodes>' argument.  For example, if you wish to run across 4 nodes, and there are 4 hosts in your mpd.host file:

mpdboot -n 4

will start the runtime daemons needed on all 4 nodes in your cluster or batch allocation.

If you are using a static list of hosts in your cluster, you create a mpd.hosts file as documented in the Configuration Set Up section above and place that in your current directory or your home directory, running mpdboot thusly:

mpdboot --file=~/mpd.hosts -n 4

for example for 4 nodes.

If you run on under PBS, the PBS batch system will create a hosts file for you with the list of hosts in your current batch allocation and set environment variable PBS_NODEFILE to point to this hosts file.  Thus, you can use this mpdboot command:

mpdboot --file=$PBS_NODEFILE -n 4

Other batch systems will similarly create a hosts file and set an appropriate environment variable.  Please consult your site system administrator for the environment variable and hosts file, as these host file conventions vary widely by batch system and local configuration.

Another mpdboot option often needed is '--rsh=[rsh | ssh]' where the remote connection method is specified.  Again, consult your site system administrator to determine if MPI uses RSH or SSH for remote connections.  In most cases ssh is used.   Thus, an example mpdboot command would look like:

mpdboot --rsh=ssh --file=~/mpd.hosts -n N
or
mpdboot --rsh=rsh --file=$PBS_NODEFILE -n N

CHECK:  run the command 'mpdtrace' or 'mpdtrace -l' to confirm that your mpdboot launched runtime daemons on each node and that each node is communicating.  If all is well, you will get a list of nodes participating in your runtime environment.  If not, you will get an error message such as:
mpdtrace: cannot connect to local mpd (/tmp/mpd2.console_rwgreen); possible causes:
1. no mpd is running on this host
2. an mpd is running but was started without a "console" (-n option)

YOU CANNOT RUN AN INTEL CAF APPLICATION UNTIL THE MPDTRACE COMMAND COMPLETES SUCCESSFULLY.


2) Run your compiled Intel CAF application

simply invoke your compiled Intel CAF application.  This application contains a job launcher that invokes the Intel MPI 'mpiexec' command using arguments from your CAF config file.

./a.out
./mycafprogram.exe

along with any command line arguments to the application.

Need to change the number of images launched or the arguments to mpiexec?  Simply change the CAF config file settings in the text file.  Remember, the -coarray-config-file=  options used at compile time set the name an location for this text file.  You should use a name and location you can remember for this file, such as -coarray-config-file=~/cafconfig.txt
Then just add mpiexec options to ~/cafconfig.txt, for example
-perhost 2 -envall -n 64 ./a.out

Note:  environment variable FORT_COARRAY_NUM_IMAGES has not effect on distributed memory CAF applications.  This environment variable is only honored by a shared memory CAF image.  Only the -n option in the CAF config file is used to control the number of CAF images for a distributed memory CAF application.

Again, read the mpiexec documentation in the Intel MPI documentation set.

3) Cleanup/shutdown:  run command 'mpdallexit' to kill the cluster runtime daemons.  This is the opposite of the 'mpdboot' command.

Known Issues or Limitations :

Many clusters have multiple MPI implementations installed along with Intel MPI.  The PATH variable, LD_LIBRARY_PATH MUST have Intel MPI paths BEFORE any other MPI installed on your system.  Make sure to ONLY source the mpivars.sh script to set this correctly OR insure that the correct Intel MPI paths appear before other MPI paths.

Batch system notes:  In the above notes, we added the option '-envall' to the CAF config file.  This is an attempt to get your current working environment variables to be inherited by your spawned remote CAF processes.  This was done to help insure that your PATH and LD_LIBRARY_PATH contain the paths to Intel MPI and Intel Fortran AND those paths appear before other MPI and compilers on your system.  HOWEVER, some batch scheduling systems will not allow environment inheritence: in other words they will throw out your current environment variables and use defaults for these.  That is why we suggested adding the 'source <path to intel MPI>/[ia32 | intel64]/bin/mpivars.sh' to your .bashrc, .cshrc, or .bash_profile.  These dot files are invoked by each child process created, and hence, SHOULD set the PATH and LD_LIBRARY_PATH appropriately.   When in doubt, execute 'which mpiexec' interactively, or put 'echo `which mpiexec`' in your batch script to insure the Intel MPI mpiexec is being used.  Other MPI implementation 'mpiexec' commands cannot be used and will cause errors.

It is critical to insure that you can execute an Intel MPI application PRIOR to attempting to run an Intel CAF program.  Keep a simple MPI 'hello world' handy to debug your environment.  Here is a sample:

program hello_mpi
implicit none
include 'mpif.h'
integer :: size, rank, ierr, len
integer :: status
character*(MPI_MAX_PROCESSOR_NAME) name

call mpi_init(ierr)
call mpi_comm_size(MPI_COMM_WORLD, size, ierr)
call mpi_comm_rank(MPI_COMM_WORLD, rank, ierr)
call MPI_GET_PROCESSOR_NAME(name, len, ierr)

write(6, "(*(a,i3))") " MPI: size = ", size, " rank = ", rank
write(6, * ) "host is ", trim(name)

call mpi_finalize(ierr)

end program hello_mpi

Compile with:   mpiifort -o hello_mpi hello_mpi.f90
run with:   mpdboot --rsh=ssh --file=~/mpd.hosts -n 2 ; mpiexec -n 4 ./hello_mpi ; mpdallexit

and different number of processes for the -n argument.  Make sure you can run across all the nodes you believe are in your cluster or your batch allocation.

READ: the Intel MPI Release Notes and the Getting_Started.pdf documents that come with Intel MPI in the <installdir>/doc/ directory.


GETTING HELP

Our User Forums are great places to see current issues and to post questions:

Intel MPI User Forum
Intel Fortran User Forum

Introduction : This article describes a method to compile and run a distributed memory coarray program using Intel® Fortran Compiler XE 12.0. An example using Linux* is presented.


Version : Intel® Fortran Compiler XE 12.0


Application Notes : To compile for distributed memory coarrays, use compiler option -coarrays=distributed (Linux* OS) or /Qcoarrays:distributed (Windows* OS).  This requires an Intel® Cluster Toolkit license.  To compile for shared memory coarrays, use compiler option -coarrays=shared (Linux* OS) or /Qcoarrays:shared (Windows* OS).  Compiling for shared memory coarrays does not require an Intel® Cluster Toolkit license.


Obtaining Source Code : The coarray example from the Composer XE 'coarray_samples' directory could be used.


Prerequisites : An Intel® Cluster Toolkit license is required for compilation, and the Intel® MPI Library must be installed on the cluster nodes.


Configuration Set Up : A key for running a distributed memory coarray program with process pinning on specific nodes is to use compiler option -coarray-config-file=filename (Linux* OS)or /Qcoarray-config-file:filename (Windows* OS).  This enables you to take full advantage of Intel® MPI Library features in the coarrays environment, in the same way that 'mpiexec -config filename' allows mpiexec to take its commands from 'filename'.

The contents of the configuration file for this example:

-host host1 -env I_MPI_PIN_PROCESSOR_LIST 0,2,4 -n 3 <path to executable>coarry_dist_host.x : -host host2 -env I_MPI_PIN_PROCESSOR_LIST 1,3,5 -n 3 <path to executable>coarry_dist_host.x

This says to execute six coarray images 'coarry_dist_host.x' on nodes host1 and host2, using processors 0,2,4 on host1, and processors 1,3,5 on host2.  The I_MPI_PIN_PROCESSOR_LIST environment variable is used to achieve the process pinning on the indicated nodes.


Source Code Changes : See Verifying Correctness


Building the Application : Compile for distributed coarrays, create one coarray image, and specify the coarray configuration file:

ifort -coarray=distributed -coarray-num-images=1 -coarray-config-file=coarray_config.txt coarry_dist_host.f90 -o coarry_dist_host.x

Running the Application : Simply specify the name of the executable:
> <path to executable>/coarry_dist_host.x
Hello from image 1 out of 6
total images, and running on host: host1

Hello from image 2 out of 6
total images, and running on host: host1

Hello from image 3 out of 6
total images, and running on host: host1

Hello from image 5 out of 6
total images, and running on host: host2

Hello from image 4 out of 6
total images, and running on host: host2

Hello from image 6 out of 6
total images, and running on host: host2
>

Verifying Correctness : Embed 'call hostnm(hostname)' in your coarray program, then print 'hostname' to verify the images are executed on the correct nodes/processors.


Benefits : This method enables coarray image pinning on specific nodes/node processors.  Better load balance across cluster nodes might be obtained, or a subset of nodes easily partitioned.


Known Issues or Limitations : 
-- Some users have reported MPI environment issues when trying to run the executable in a standalone fashion.  These issues are under investigation, but as a workaround try using mpiexec to launch the executable. 
--Distributed memory coarrays only work with Intel® MPI; other implementations of MPI are not supported.

Operating System :
Red Hat Enterprise Linux* 5
SuSE Linux Enterprise Server* 10


Problem Description :
MKL experiences the performance degradation in the case dgemm when MKL works on
on SGI Altix systems with Nehalem-EX CPU.
The cause of the problem is that MKL wrong detects number of threads which are available on this type of system


Resolution Status :
The problem has been fixed and is available in the versions of Intel® MKL 10.2 Update 6 and later versions.


[DISCLAIMER: The information on this web site is intended for hardware system manufacturers and software developers. Intel does not warrant the accuracy, completeness or utility of any information on this site. Intel may make changes to the information or the site at any time without notice. Intel makes no commitment to update the information at this site. ALL INFORMATION PROVIDED ON THIS WEBSITE IS PROVIDED "as is" without any express, implied, or statutory warranty of any kind including but not limited to warranties of merchantability, non-infringement of intellectual property, or fitness for any particular purpose. Independent companies manufacture the third-party products that are mentioned on this site. Intel is not responsible for the quality or performance of third-party products and makes no representation or warranty regarding such products. The third-party supplier remains solely responsible for the design, manufacture, sale and functionality of its products. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others.]