I have a master/slave type MPI program, and I'd like the master to dynamically spawn the slave processes. I tried MPI_Comm_spawn, but it seems that I could only start slave processes on nodes where mpd.py has already been started, ie., nodes specified in mpd.hosts. However, in my case, I'd like to assume that I don't know which nodes I will use when I start the program using mpirun. The nodes where the slave processes will run are determined at run-time.
I tried to use the hydra process manager in Intel 4.1, and MPI_Comm_spawn failed. Does hydra support spawning at all?
Could anyone give me some insight or advice on how to solve my problem? Thanks.