Remote Terminal MPI Debug using Intel® Debugger


Introduction : 

By default the Intel® Debugger requires the ability to open either a GUI or at least a dedictaed separate xterm window. This article describes how to launch a command line verion of MPI enabled IDB in a command line only terminal window.


Version :
This article applies to the Intel® Debugger for Linux* v11.0 and higher
The proposed command ine and testing of this approach was based on Intel® MPI Library 3.x, but newer versions should work as well.
The screen command needs to be available

Configuration Set Up :
The usage model this article applies to is as follows:

- network based Linux MPI cluster with application possibly running distributed over several nodes

- only available access to the cluster is via remote terminal login (puTTY, ssh, ...)

- no X window output redirection to the login machine is possible 

- Please ensure to IDB_PARALLEL_SHELL set to enable multi-node debugging

"export IDB_PARALLEL_SHELL=/usr/bin/ssh"

Solution:

- Edit mpiexec.py launch script in the bin directory of your Intel® MPI Library installation
- Launch debugger with "screen mpiexec -idb -n <node number> <debuggee executable>"

Python Script Changes :

1. In the line that defines dbg_cmd replace "xterm" with "screen" and remove "-e" in mpiexec.py. In addition remove the "&" at the end of the line.

E.g. replace the line

dbg_cmd = 'xterm -e ' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python &'

with

dbg_cmd = 'screen' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python'

2. Replace the line

os.system(dbg_cmd)

with the following lines:

cpid = os.fork()

if (cpid == 0):
os.system(dbg_cmd)
sys.exit(0)

For more complete information about compiler optimizations, see our Optimization Notice.

3 comments

Top
Maxim Sorockin's picture

Hi Rob.

I changed mpiexec.py script but it still don't works.
If I run

screen mpiexec -idb -n <node number> <debuggee executable>

I obtain

[screen is terminating]

for

mpiexec -idb -n <node number> <debuggee executable>

I have

mpiexec_cluster.immsp.kiev.ua: mpd_uncaught_except_tb handling:
<type 'exceptions.ImportError'>: /opt/intel/impi/3.2.0.011/bin64/mtv.so: undefined symbol: Py_InitModule4
/opt/intel/impi/3.2.0.011/bin64/mpiexec 1005 mpiexec
import mtv
/opt/intel/impi/3.2.0.011/bin64/mpiexec 2772 <module>
mpiexec()

Maxim

anonymous's picture

The article has been updated. Everything works fine right now (4.0 as well).
--Dmitry

anonymous's picture

Hi Robert,
You probably created this article for Intel(r) MPI Library only because MPICH's mpiexec doesn't support '-idb' option.
I should mention also that 'screen' is not always available, e.g. not all our servers have this utility installed.
I've tried you suggestion with 64-bit version of the IDB from compiler 11.1 with Intel(r) MPI version 4.0.0.025 and got "Must be connected to a terminal" error, but original xterm version works just fine. Might be there some other settings you missed?

It's quite interesting approach and I'm ready to discuss it via e-mail.
---Dmitry

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.