I am using Intel(R) MPI Library for Linux* OS, Version 4.1 Update 1 Build 20130522 on a Linux Cluster environment. Running the following script will produce a race condition. All used libraries are compiled against this MPI library.
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() world_group = comm.Get_group() my_group = world_group.Incl([rank]) my_comm = comm.Create(my_group) intercomm = my_comm.Spawn("./script.sh", , 1, MPI.INFO_NULL, 0)
Randomly occuring error:
[mpiexec@capp1] control_cb (./pm/pmiserv/pmiserv_cb.c:715): assert (!closed) failed
[mpiexec@capp1] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@capp1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:430): error waiting for event
[mpiexec@capp1] main (./ui/mpich/mpiexec.c:847): process manager error waiting for completion
A possible workaround is given by serialization of the MPI_Comm_spawn calls:
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() size = comm.Get_size() world_group = comm.Get_group() my_group = world_group.Incl([rank]) my_comm = comm.Create(my_group) for r in range(0, size): if rank == r: intercomm = my_comm.Spawn("script.sh", , 1, MPI.INFO_NULL, 0) comm.Barrier()
It will also dissappear if comm is used to call MPI_Comm_spawn collectively, which will also induce mandatory serialization. A completely parallel call is not reliably possible.
It seems to me that the process manager is not multi process safe in this case. In my opionion and understanding this use scenario should be possible according to the MPI standard. I tested this behaviour for the shm and rdma fabric.
Thank you very much for your input!