I have a program, test.f90, that uses system() calls and am trying to instrument it using the itcpin utility as part of the Intel® Trace Analyzer and Collector. Compilation is as follows:

$ mpiifort -o test.exe test.f90

I'm able to run it without issues:

$ mpiexec -n 2 ./test.exe
Process 1 of 2 proc on host node1
Process 2 of 2 proc on host node2

Manually linking with the Intel® Trace Collector library functions correctly and produces trace files:

$ mpiexec -trace -n 2 ./test.exe
Process 1 of 2 proc on host node1
Process 2 of 2 proc on host node2
[0] Intel(R) Trace Collector INFO: Writing tracefile test.exe.stf in /home/user

But using binary instrumentation produces the following error:

$ mpiexec -n 2 itcpin --run -- ./test.exe
forrtl: severe (24): end-of-file during read, unit 20, file /tmp/mhost_0
Warning: execution of command "
/bin/hostname > /tmp/mhost_0
" failed
rank 0 in job 15 node1_54486 caused collective abort of all ranks
exit status of rank 0: return code 24

Furthermore, the error message at failure is:

DPLACE BUG: libdplace: dplace_ioctl: Invalid argument

What's wrong?

Solution:

The issue lies in the system() call used and the fact that it forks some number of children. In this case, itcpin follows the forks but, because they're not MPI processes, is unable to determine the correct default VT library to insert. The current work-around is to manually insert libVT by using the --insert option as follows:

$ mpiexec -n 2 itcpin --run --insert VT -- ./test.exe