I'm trying to run the Intel® MPI Library on my Ubuntu system. Installation went fine but I see this error when trying to run:

gsslavov@backuppc:~$ mpirun -np 4 hostname
/home/gsslavov/intel/impi/3.2.2.006/bin/mpirun: 63: Syntax error: "(" unexpected (expecting "fi")

What's going on?

Environment : 

This applies to systems running Ubuntu 9.10 or later (as well as Ubuntu 8.04 LTS or later)

Root Cause : 

It seems that with version 9.10, Ubuntu has decided to change the default shell from bash to dash. More info is available from the link in the excerpt below:

In Ubuntu 6.10, the default system shell, /bin/sh, was changed to dash (the Debian Almquist Shell); previously it had been bash (the GNU Bourne-Again Shell). This document explains this change and what you should do if you encounter problems.

The default login shell remains bash.

The two shells are not fully compatible (since bash is quite more extensive than dash), thus leading to the syntax errors.

Resolution : 

If you'd like to continue using the Intel MPI Library (or any of the Cluster Tools) on Ubuntu, the fix for this issue is fairly simple: change the /bin/sh symlink to point from dash to bash:

$ rm /bin/sh
$ ln -s /bin/bash /bin/sh

The Intel® MPI Library fails intermittently when run over the RDSSM or RDMA devices. Approximately 5-10% of runs fail on RHEL (Red Hat Enterprise Linux) 5.4, but this problem does not occur on earlier versions of RHEL.

When reviewing the debug output, the following error is seen during the Intel MPI Library operations:

setup_listener Cannot assign requested address

Environment:

Red Hat Enterprise Linux 5.4 only

Root Cause:

This error occurs with the Intel MPI Library and the version of OFED (Open Fabrics Enterprise Distribution) included with RHEL 5.4. There is a potential port space conflict with RDS (reliable datagram sockets) and when this port space conflict occurs, uDAPL does not resolve it correctly.

By default, the Intel MPI Library uses its process ID to define its port number. In RHEL 5.4, the process ID can occasionally match a port number that the RDS driver has already allocated, which creates a port space conflict. Currently, uDAPL will reply with the wrong return code to the Intel MPI Library and communication will fail.

Resolution:

As a temporary workaround, set the following environment variable on all nodes:

$ export I_MPI_RDMA_CREATE_CONN_QUAL = 0

After setting this variable, the Intel MPI Library will not define its port number from its process ID.

This error is resolved in DAPL 2.0.25, to be included in Open Fabrics Enterprise Distribution (OFED) 1.5.  Status of the resolution can be found in the latest OFED release notes.

ERROR: could not determine icc directory for driver: mpiifort

We are not using the Intel® C Compiler on our cluster system, nor does this error affect any of the other MPI compilation drivers. The Intel® Fortran compiler works fine on its own. How can I fix this issue with Intel MPI Fortran compilation?

The mpiifort compilation script calls a shell script named compchk.sh. A short term fix is to compile the application test.f90 or test.f with -nocompchk as follows:

mpiifort -nocompchk test.f90
and
mpiifort -nocompchk test.f

Note: This issue appears for older versions of the Intel® MPI Library (1.x to 2.x). It is strongly encouraged that you upgrade to the latest version of the Intel MPI Library instead. This check has been disabled in later releases.

Operating System:

When compiling certain C++ applications using the Intel® MPI Library compilation driver mpiicpc, I encounter an error message that looks something like the following:

…
/opt/intel/mpi/3.1/include/mpicxx.h(45): catastrophic error: #error directive: "SEEK_SET is #defined but must not be for the C++ binding of MPI"
#error "SEEK_SET is #defined but must not be for the C++ binding of MPI"
^ 

Compilation aborted for my_mpi_program.cpp (code 4)
make: *** [my_mpi_program.o] Error 4

For the application my_mpi_program.cpp, you have two options:

1. Try organizing the include files such that mpi.h comes before stdio.h
2. Use the -DMPICH_IGNORE_CXX_SEEK and -DMPICH_SKIP_MPICXX options at compile time
   • Refer to http://www-unix.mcs.anl.gov/mpi/mpich/faq.htm#cxxseek†  for more information.

^† This link will take you off of the Intel Web site. Intel does not control the content of the destination Web Site.

I have a cluster, which has for example 3 nodes, where the mpd.hosts file contains:

$ cat mpd.hosts
node2
node3

The master node, which is node1 is not listed in the mpd.hosts file. The shell commands:

$ mpdboot -r ssh -n 2 -f ~/mpd.hosts
$ mpiexec -n 4 /bin/hostname

node1
node1
node2
node2

Solution

By design, the master node (i.e., the node where the mpiexec command is launched from) needs to have a daemon running. If you want to run your application on node2 and node3, you can use the -host options with the mpiexec command as follows:

$ mpdboot -r ssh -n 3 -f ~/mpd.hosts
$ mpiexec -n 2 -host node2 /bin/hostname : -n 2 -host node3 /bin/hostname

$ mpdboot -r ssh -n 3 -f ~/mpd.hosts
$ mpiexec -nolocal -n 4 /bin/hostname

This will run the hostname command on nodes node2 and node3 only, even though there are three mpd daemons total.