• 2019 Update 3
  • 03/07/2019
  • Public Content
  • Download as PDF
Contents

Normally, if an MPI application fails or is aborted, all the trace data collected is lost, because
libVT
needs a working MPI to write the trace file. However, the user might want to use the data collected up to that point. To solve this problem, Intel® Trace Collector and Analyzer provides the
libVTfs
library that enables tracing of failing MPI applications.
Usage Instructions
To trace failing MPI applications, do the following:
Linux* OS
Set the
LD_PRELOAD
environment variable to point to the
libVTfs
library and run the application. For example:
$ export LD_PRELOAD=libVTfs.so $ mpirun -n 4 ./myApp
Alternatively, rebuild your application with the static version of the library. For example:
$ mpiicc -profile=vtfs myApp.c -o myApp
Windows* OS
Relink your application with the
libVTfs
library before the MPI library and run it as usual. To do this, you should create an Intel® MPI Library configuration file that points to the
libVTfs
library. You can do it as follows (administrator privileges may be required):
> echo SET PROFILE_PRELIB=%VT_ROOT%\lib\VTfs.lib > %I_MPI_ROOT%\intel64\lib\VTfs.conf > mpiicc -profile=VTfs myApp.c > mpiexec -n 4 myApp.exe
How it Works
Under normal circumstances tracing works like with
libVT
, but communication during trace file writing is done through TCP sockets, so it may take more time than over MPI. In order to establish communication, it needs to know the IP addresses of all the hosts involved. It finds them by looking up the hostname locally on each machine or, if that only yields the 127.0.0.1 local host IP address, falls back to broadcasting hostnames. In the latter case hostname lookup must work consistently in the cluster. In case of a failure,
libVTfs
freezes all MPI processes and then writes a trace file with all trace data.
Possible Failures
Failure
Description
Signals
Includes events inside the application like segmentation faults and floating point errors, and also abort signals sent from outside, like
SIGINT
or
SIGTERM
.
Only
SIGKILL
will abort the application without writing a trace because it cannot be caught.
Premature Exit
One or more processes exit without calling
MPI_Finalize()
.
MPI Errors
MPI detects certain errors itself, like communication problems or invalid parameters for MPI functions.
Deadlocks
If Intel® Trace Collector observes no progress for a certain amount of time in any process, it assumes that a deadlock has occurred, stops the application and writes a trace file.
You can configure the timeout with
DEADLOCK-TIMEOUT
. "No progress" is defined as "inside the same MPI call". This is only a heuristic and may fail to lead to both false positives and false negatives.
Undetected Deadlock
If the application polls for a message that cannot arrive with
MPI_Test()
or a similar non-blocking function, Intel® Trace Collector still assumes that progress is made and does not stop the application.
To avoid this, use blocking MPI calls in the application, which is also better for performance.
Premature Abort
If all processes remain in MPI for a long time due to a long data transfer for instance, then the timeout might be reached. Because the default timeout is five minutes, this is very unlikely. After writing the trace
libVTfs
will try to clean up the MPI application run by sending all processes in the same process group an
INT
signal. This is necessary because certain versions of MPICH* may have spawned child processes which keep running when an application aborts prematurely, but there is a certain risk that the invoking shell also receives this signal and also terminates. If that happens, then it helps to invoke
mpirun
inside a remote shell:
rsh localhost 'sh -c "mpirun . . . "'
MPI errors cannot be ignored by installing an error handler.
libVTfs
overrides all requests to install one and uses its own handler instead. This handler stops the application and writes a trace without trying to proceed, otherwise it would be impossible to guarantee that any trace will be written at all.
On Windows* OS, not all features of POSIX* signal handling are available. Therefore,
VTfs
on Windows* OS uses some heuristics and may not work as reliably as on Linux* OS. It is not possible to stop a Windows* application run and get a trace file by sending a signal or terminating the job in the Windows task manager.

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804