MPI program behavior on node crash ...

MPI program behavior on node crash ...

Na Na's picture
Na Na
Sun, 07/29/2012 - 18:38

In the production environment, it happens that some nodes crash once in a while. What's the behavior of Intel's MPI when an MPI program encounters lost contact of some of its processes? Would there be any difference if the node crashed contains rank 0? Is there any option of Intel's MPI to control the behavior of such situation so that the program will be cleaned up in case one of the MPI processesis lost?Thank you very much,Tofu

RSS Top
2 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.
James Tullos (Intel)'s picture
James Tullos (Intel)
Mon, 07/30/2012 - 07:34

Hi Tofu,

If a node containing a process crashes, the entire job will end. You can use the -cleanup option (or I_MPI_HYDRA_CLEANUP) to create a temporary file that will list the PID of each process, and the mpicleanup utility will use this file to clean the environment if the job does not end correctly. You can also use I_MPI_MPIRUN_CLEANUP if you are using MPD instead of Hydra.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Top

Login to leave a comment.