Introduction : 

By default the Intel(R) Debugger requires the ability to open either a GUI or at least a dedictaed separate xterm window. This article describes how to launch a command line verion of MPI enabled IDB in a command line only terminal window.


Version :
This article applies to the Intel(R) Debugger for Linux* v11.0 and higher
The proposed command ine and testing of this approach was based on Intel(R) MPI Library 3.x, but newer versions should work as well.
The screen command needs to be available

Configuration Set Up :
The usage model this article applies to is as follows:

- network based Linux MPI cluster with application possibly running distributed over several nodes

- only available access to the cluster is via remote terminal login (puTTY, ssh, ...)

- no X window output redirection to the login machine is possible 

- Please ensure to IDB_PARALLEL_SHELL set to enable multi-node debugging

"export IDB_PARALLEL_SHELL=/usr/bin/ssh"

Solution:

- Edit mpiexec.py launch script in the bin directory of your Intel(R) MPI Library installation
- Launch debugger with "screen mpiexec -idb -n <node number> <debuggee executable>"

Python Script Changes :

1. In the line that defines dbg_cmd replace "xterm" with "screen" and remove "-e" in mpiexec.py. In addition remove the "&" at the end of the line.

E.g. replace the line

dbg_cmd = 'xterm -e ' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python &'

with

dbg_cmd = 'screen' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python'

2. Replace the line

os.system(dbg_cmd)

with the following lines:

cpid = os.fork()

if (cpid == 0):
os.system(dbg_cmd)
sys.exit(0)

It’s as easy as 1-2-3:

1. Create a tmi.conf file for your chosen fabric driver. You can find an example tmi.conf file in the <impi_install_dir>/etc[64] directory.
   • If your cluster is running the Myrinet* MX* drivers, your tmi.conf file should have the following contents:

     $ cat tmi.conf
     # TMI provider configuration
     mx 1.0 libtmip_mx.so " " # comments ok

   • If your cluster is running the Qlogic* PSM* drivers, your tmi.conf file should have the following contents:

     $ cat tmi.conf
     # TMI provider configuration
     psm 1.0 libtmip_psm.so " " # comments ok

   Make sure your LD_LIBRARY_PATH includes the Intel MPI Library lib[64] directory. You can do that easily by sourcing the provided mpivars.[c]sh scripts.

   Alternatively, you can specify the full path to the TMI library in the tmi.conf file as illustrated in the following example for the PSM* interface:

   $ cat tmi.conf
   # TMI provider configuration
   psm 1.0 /opt/intel/impi/4.0/lib64/libtmip_psm.so " " # comments ok

    

2. Position the new tmi.conf file appropriately. The default location for this configuration file is /etc/tmi.conf. If the /etc directory is not shared amongst all nodes, manually copy the tmi.conf file across the cluster.

   Alternatively, you can use the TMI_CONFIG environment variable to point to a new location, as follows:

   $ export TMI_CONFIG=/home/myid/tmi.conf

   If you use the TMI_CONFIG variable as specified, the Intel MPI Library will use the new /home/myid/tmi.conf configuration file instead of /etc/tmi.conf.

    

3. Enable the use of the tmi fabric on your Intel MPI Library command line by setting the I_MPI_FABRICS environment variable as follows:

   $ export I_MPI_FABRICS=shm:tmi # for shared memory and TMI communication

   or

   $ export I_MPI_FABRICS=tmi # for TMI-only communication

   Alternatively, this can be selected at runtime on your command line:

   $ mpirun –genv I_MPI_FABRICS shm:tmi –n 2 ./exe

    

You can find additional information in the Intel MPI Library Reference Manual, or contact us by posting at the Intel® Clusters and HPC Technology Forums.

1. Determine if a sockets interface is provided. If yes, the interconnect will work with TCP/IP performance out of the box.
2. If no sockets interface is available, or if you’d like to enable maximum interconnect performance and functionality, a fabric provider layer must be developed. As an interconnect vendor, you have the following options:
   • Direct Access Programming Library (DAPL*) provider
   • OpenFabrics Alliance* (OFA*) verbs provider
   • Tag Matching Interface (TMI) provider
   All three APIs are used to drive RDMA (remote direct memory access)-capable fabrics like InfiniBand*, Myrinet* MX*, and Qlogic* PSM*.

What steps should be performed to build your own Direct Access Programming Library (DAPL*) provider?

1. Obtain the DAPL software by visiting the SourceForge repository.
2. Visit the DAT Collaborative website for additional information on DAPL or to check out other InfiniBand* DAPL implementations. You can also join the OpenFabrics Alliance* for access to even more resources.
3. Decide on a level of DAPL to support. The suggested level is 2.0.
4. Develop your DAPL driver. You can use the Intel MPI Library Fabric Provider writer guide for hints and ideas. The document is available free of charge as long as you are willing to sign an Intel-standard Non-Disclosure Agreement (NDA).
5. Test the DAPL layer with the DAPL test suite that comes with the DAPL source code.
6. Obtain the Intel MPI Library. The software can be purchased from the Intel website by visiting www.intel.com/go/mpi.
7. Link with the Intel MPI Library and test with the Intel MPI Library test suite. The Intel MPI Library test suite is provided upon request at the Intel® Premier Web Portal.

What steps should be performed to build your own OpenFabrics Alliance (OFA*) verbs provider?

1. Obtain the OFA verbs software by visiting the OpenFabrics download page.
2. Visit the OpenFabrics Alliance* website for additional information on the verbs or to join the alliance.
3. Develop your OFA verbs driver. You can use the Intel MPI Library Fabric Provider writer guide for hints and ideas. The document is available free of charge as long as you are willing to sign an Intel-standard NDA.
4. Obtain the Intel MPI Library. The software can be purchased from the Intel website by visiting www.intel.com/go/mpi.
5. Link with the Intel MPI Library and test with the Intel MPI Library test suite. The Intel MPI Library test suite is provided upon request at the Intel® Premier Web Portal.

What steps should be performed to build your own Tag Matching Interface (TMI) provider?

1. Obtain the TMI API by submitting a request through the Intel® Premier Web Portal. You need to sign an NDA with Intel before we can provide you the document.
2. Develop your TMI driver. You can use the Intel MPI Library Fabric Provider writer guide for hints and ideas. The document is available free of charge upon signing the Intel-standard NDA.
3. Obtain the Intel MPI Library. The software can be purchased from the Intel website by visiting www.intel.com/go/mpi.
4. Link with the Intel MPI Library and test with the Intel MPI Library test suite. The Intel MPI Library test suite is provided upon request at the Intel® Premier Web Portal.

    double alpha = 1.0; double beta = 0.0;
    pdgemv_("N",&M,&M,&alpha,A,&ONE,&ONE,descA,x,&ONE,&ONE,descx,&ONE,&beta,y,&ONE,& amp;ONE,descy,&ONE);
 
3. Build and Run it
    Build it under linux environment, assume intel 64bit application. 
    ! set environment and build pdgemv.c 
    $source /opt/intel/mkl/10.x.x.0xx/tools/environment/mklvarsem64t.sh
    $source /opt/intel/mpi/3.x.x/bin64/mpivars.sh
    $source /opt/intel/Compiler/11.x/0xx/bin/iccvars.sh intel64
    $mpiicc -w -o pdgemv pdgemv.c -I/opt/intel/mkl/10.x.x.0xx/include -L/opt/intel/mkl/10.x.x.0xx/lib/em64t
     -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread

    ! run the application 
    $ mpirun -n 4 ./pdgemv

4. Notes
Intel MKL PBLAS/ScaLAPACK routines are written in FORTRAN interface, so Column-major are used and row-major is not acceptable.  If you gives the matrix to these routines in row-major - even your code is completely right from C (Row-major) point of view,  the result will be almost always wrong.

I'm trying to run the Intel® MPI Library on my Ubuntu system. Installation went fine but I see this error when trying to run:

gsslavov@backuppc:~$ mpirun -np 4 hostname
/home/gsslavov/intel/impi/3.2.2.006/bin/mpirun: 63: Syntax error: "(" unexpected (expecting "fi")

What's going on?

Environment : 

This applies to systems running Ubuntu 9.10 or later (as well as Ubuntu 8.04 LTS or later)

Root Cause : 

It seems that with version 9.10, Ubuntu has decided to change the default shell from bash to dash. More info is available from the link in the excerpt below:

In Ubuntu 6.10, the default system shell, /bin/sh, was changed to dash (the Debian Almquist Shell); previously it had been bash (the GNU Bourne-Again Shell). This document explains this change and what you should do if you encounter problems.

The default login shell remains bash.

The two shells are not fully compatible (since bash is quite more extensive than dash), thus leading to the syntax errors.

Resolution : 

If you'd like to continue using the Intel MPI Library (or any of the Cluster Tools) on Ubuntu, the fix for this issue is fairly simple: change the /bin/sh symlink to point from dash to bash:

$ rm /bin/sh
$ ln -s /bin/bash /bin/sh

Officially, the Intel Cluster Tools support RPM-based installs only. For a full list of supported OSes, check out our compatibility matrix.

If you’d still like to use the tools on Debian and Ubuntu systems, the following steps can help you in, at least, installing the tools. Note that the process below still requires that a valid license for the desired Cluster Tool is present on the machine.

1. Download the latest version of the Intel® MPI Library for Linux* from the Intel® Registration Center.


2. Untar the package and go down into the newly created directory

   $ tar –xzf l_mpi_p_3.2.1.009.tgz
   $ cd l_mpi_p_3.2.1.009/



3. Create a new directory and extract the RPM files using the provided install.sh script. Note that you’d need a valid license to do so.

   $ mkdir rpms
   
   $ ./install.sh --extract=./rpms
   Extracting components...
   Extracting files... ################################################ [100%]
   Product's rpm and EULA have been extracted in ./rpms
   Please carefully read EULA before using the product
   
   $ ls –l
   

   intel-mpi-em64t-3.2.1p-009.x86_64.rpm	# SDK for the Intel MPI Library for Linux*
   intel-mpi-rt-em64t-3.2.1p-009.x86_64.rpm	# Runtime component of the Intel MPI Library
   mpiEULA.txt	# End-User License Agreement for the Intel MPI Library. # Read before using.
   redist.txt	# List of Intel MPI Library files whose redistribution is allowed



4. Convert the RPM files into DEB files, to be installed on your Debian-based system using the alien utility

   $ which alien
   /usr/bin/alien
   
   $ alien intel-mpi-em64t-3.2.1p-009.x86_64.rpm
   intel-mpi-em64t_3.2.1p-10_amd64.deb generated

   Here note that converting the SDK package for the Intel MPI Library is enough, as the runtime libraries are already contained within.


5. Now that you have the .deb file generated, install using your preferred method. For example, this is how it’s done using the dpkg utility

   # This will use the default /opt/intel installation directory
   $ dpkg --install intel-mpi-em64t_3.2.1p-10_amd64.deb



6. Done! You are now free to enjoy the Intel MPI Library at your leisure.



Even though this example highlighted the Debian-based installation of the Intel MPI Library, the same technique can be applied to the Intel® Cluster Toolkit for Linux*, Intel® Cluster Toolkit Compiler Edition for Linux*, and the Intel® Trace Analyzer and Collector for Linux*.

NOTE: It seems like Ubuntu has changed the default system shell starting with version 9.10. If you're seeing syntax errors, the following article talks about how they can be resolved.

The below instructions describe how to run Intel MPI Library jobs using Sun Grid Engine. This document relates to Linux*. While there are some differences and additional steps when using Microsoft* Windows*, in general the procedure is the same.

All optional steps are recommended but not necessary for successful integration.

1. [Optional] Visit sun.com and get a brief overview of SGE
2. Installation

   See the Installation Guide from sun.com for details. Roughly, the steps are as follows:

   • Install Master Host (see ‘How to Install the MasterHost’ section);
   • Install Execution Host (see ‘How to Install ExecutionHosts’ section);
   • Register Administration Hosts (see the corresponding section in the Installation Guide);
   • Register Submit Hosts (see corresponding section);
   • Verify the installation (see corresponding section).

   IMPORTANT NOTES:

   • To finalize the installation process, you’ll have to configure the network services manually (by modifying /etc/services), which requires root privileges.
   • It’s possible to install/run SGE as a non-privileged user, but
     1. there are some limitations in that case;
     2. you need root privileges for the complete installation process (at least, for modifying /etc/services).
3. Create a new Parallel Environment (PE) for Intel MPI Library
   
   1. Create the appropriate configuration file for the new PE. It should contain the following lines:

      pe_name	impi
      slots	999
      user_lists	NONE
      xuser_lists	NONE
      start_proc_args	NONE
      stop_proc_args	NONE
      allocation_rule	$round_robin
      control_slaves	FALSE
      job_is_first_task	FALSE
      urgency_slots	min

   2. Add the new PE using the following command:

      ‘qconf –Ap <config_file>’

   USEFUL COMMANDS:
   * qconf –spl – view all PEs currently available;
   * qconf –sp <PE_name> - view settings for a particular PE;
   * qconf –dp <PE_name> - remove a PE;
   * qconf –mp <PE_name> - modify an existing PE.

   Also see the ‘Managing Special Environment’ section in the Administration Guide from sun.com if you need more details about PE configuration.

4. Associate a queue with the new PE

   Use the following commands for that:

   1. qconf –sql – to see all queues available;
   2. qconf –mq <queue_name> - to modify the queue’s settings. Find the ‘pe_list’ property in the open window and add the ‘impi’ string to that property.

   USEFUL COMMANDS:
   * qconf –sq <queue_name> - view the queue’s settings.

   See the Administration Guide if you need more details about the queue configuration process.

5. Add Intel MPI Library environment to your current environment by sourcing the appropriate mpivars.[c]sh script located in the <install_dir>/bin[64] directory
6. Build the MPI application to be run
7. [Optional] Make sure that Intel MPI Library works fine on the desired hosts. For this, manually run your application on the desired hosts individually
8. Submit your MPI job to SGE

   Use the following command for that:

   qsub -N <job_name> -pe impi <num_of_processes> \
-V <mpirun_absolute_name> -r ssh -np <num_of_processes> <app_absolute_name>

   
   where
   -V option is used so that all environment variables available in the current shell are exported to a job.

    

   USEFUL COMMANDS to monitor and control jobs:
   * qstat – show status of SGE jobs and queues;
   * qstat –j – show detailed information about jobs (can be useful for pending jobs);
   * qdel – remove existing job.
   After submitting the job you can monitor its status using the qstat command. When the job is finished, you can find the job’s output and error output in your HOME directory – just look for <job_name>.o<jobID> and <job_name>.e<jobID> files.

   See the User’s Guide, if you need more information about the job submission process.

The Intel® MPI Library fails intermittently when run over the RDSSM or RDMA devices. Approximately 5-10% of runs fail on RHEL (Red Hat Enterprise Linux) 5.4, but this problem does not occur on earlier versions of RHEL.

When reviewing the debug output, the following error is seen during the Intel MPI Library operations:

setup_listener Cannot assign requested address

Environment:

Red Hat Enterprise Linux 5.4 only

Root Cause:

This error occurs with the Intel MPI Library and the version of OFED (Open Fabrics Enterprise Distribution) included with RHEL 5.4. There is a potential port space conflict with RDS (reliable datagram sockets) and when this port space conflict occurs, uDAPL does not resolve it correctly.

By default, the Intel MPI Library uses its process ID to define its port number. In RHEL 5.4, the process ID can occasionally match a port number that the RDS driver has already allocated, which creates a port space conflict. Currently, uDAPL will reply with the wrong return code to the Intel MPI Library and communication will fail.

Resolution:

As a temporary workaround, set the following environment variable on all nodes:

$ export I_MPI_RDMA_CREATE_CONN_QUAL = 0

After setting this variable, the Intel MPI Library will not define its port number from its process ID.

This error is resolved in DAPL 2.0.25, to be included in Open Fabrics Enterprise Distribution (OFED) 1.5.  Status of the resolution can be found in the latest OFED release notes.