Hi Eric,

You are actually looking at two different capabilities. The first is a job scheduler. In Windows*, the Intel MPI Library integrates with the Microsoft* Job Scheduler and the PBS Pro* Job Scheduler. If you are using one of those, you would use the instructions in the Reference Manual. These schedulers still call mpiexec to launch the processes.

It sounds like you are using a different scheduler. In this case, what I would recommend is to set up a wrapper that will generate a configuration file and/or set appropriate environment variables for each MPI run, and then call mpiexec with that configuration file or environment. Are you using a custom scheduler, or is it one that is available to the public?

The second capability is process management within an MPI job. The MPI_Comm_spawn and other MPI-2 process management capabilities are supported in the Intel MPI Library. You will still need to launch the original executable with mpiexec in order to use this capability.

Generally, you can launch an MPI application without using mpiexec, but then it will only run as a single instance. Typically with MPI applications, each instance should be linked to the others under mpiexec. Process spawning is still handled under mpiexec.

Please let me know if you have additional questions or need further detail or clarification.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi Eric,

What is your MPI application trying to do that would require a visible desktop on nodes other than the launching node? Typically, I/O (and any user interaction) should only occur from a single rank in the job, and if using standard input and output, will be redirected to the computer that launched the job initially, even if none of the processes are running on that computer.

While the standard allows two different communicators to connect to each other, this is something I've never tried. I believe that they will still need to be launched from mpiexec to obtain the proper communications, as this is the case with other MPI commands that involved process management (I have personally tried this with MPI_Comm_spawn).

The process rank and communicator size are set and provided by mpiexec, not SMPD. SMPD only exists as a means to launch the processes ona remote host.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi Eric,

As I stated before, in order to call MPI_Comm_spawn, you will need to run under mpiexec (with the Intel MPI Library). Trying to run without mpiexec and calling MPI_Comm_spawn leads to an application crash.

Starting outside of mpiexec, the process will have a rank of 0 and size of 1. Keep in mind that rank and size are communicator specific. If you use MPI_Comm_spawn to create new processes, the new processes will have a different MPI_COMM_WORLD from the parent. The ranks are not unique either. If a parent with rank 0 spawns 3 child processes, the first child process will also have rank 0 within the new communicator. If you build a communicator that has the parent and the children, the rank within that communicator will depend on the order the processes are added.

Doesyour application interact with the user in any way? Is it a single GUI for all instances, or a separate GUI for each instance?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi Eric,

I have not yet tested that scenario, so I cannot say for certain whether or not it would work. However, launching without mpiexec will give each individual instance a rank of 0 and size of 1 (for MPI_COMM_WORLD). When an application joins using MPI_Comm_accept/MPI_Comm_connect, new communicators are created.

For example, let's say you start four instances, one acting as server and the other three joining as clients. When the first client connects to the server, the server would now have two communicators, MPI_COMM_WORLD(unchanged), and client_comm1(intercommunicator with the server as the local group and the client as the remote group). The client would also have two communicators, MPI_COMM_WORLD(unchanged, but not the same as the MPI_COMM_WORLD from the server), and server_newcomm(intercommunicator with the client as the local group and the server as the remote group). The same would be true for each of the other clients, with a new communicator in the server (client_comm2, client_comm3) for each new client.

At this point, the clients can communicate with the server, and the server can communicate with the clients. However, the clients cannot directly communicate with each other. In order to do this, you would need to build a new communicator that contains all of the instances (or at least all of the clients, if you don't want the server involved). In this communicator, each client will have a rank, and the size would be the size of the new communicator.

I believe this is what you are wanting. If so, then it should work, but as I said, I have not tested this and cannot confirm if it does work. If it does not work within the Intel MPI Library, then it is something that could be added to a future release.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi Eric,

As an update, I have just tested using MPI_Comm_accept and MPI_Comm_connect in Windows*, and it is possible to connect to a server application without using mpiexec.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi Eric,
I made an independent scheduler to manage MPI_COMM_Spawn processes to optimize the cluster resources but i have a problem while i want to runfibonnaciprogram that uses MPI_COMM_Spawn in it like recursion program but the new process will be spawned in another machine according to my scheduler but i have problem when i tried to spawn more than 8 procesess it gives me this problem : Fatal error in MPI_Comm_spawn: Other MPI error, error stack: MPI_Comm_spawn(145).............................: MPI_Comm_spawn(cmd="./fibonacci", argv=0xbfad8fc4, maxprocs=1, info=0x9c000002, root=0, MPI_COMM_SELF, intercomm=0xbfad8e60, errors=0xbfad8e64) failed MPIDI_Comm_spawn_multiple(230)..................: MPID_Comm_accept(153)...........................: MPIDI_Comm_accept(937)..........................: MPIDI_Create_inter_root_communicator_accept(205): MPIDI_CH3I_Progress(150)........................: MPID_nem_mpich2_blocking_recv(948)..............: MPID_nem_tcp_connpoll(1720).....................: state_commrdy_handler(1556).....................: MPID_nem_tcp_recv_handler(1446).................: socket closed

Fatal error in MPI_Comm_spawn: Other MPI error, error stack:MPI_Comm_spawn(145).............................: MPI_Comm_spawn(cmd="./fibonacci", argv=0xbfad8fc4, maxprocs=1, info=0x9c000002, root=0, MPI_COMM_SELF, intercomm=0xbfad8e60, errors=0xbfad8e64) failedMPIDI_Comm_spawn_multiple(230)..................:MPID_Comm_accept(153)...........................:MPIDI_Comm_accept(937)..........................:MPIDI_Create_inter_root_communicator_accept(205):MPIDI_CH3I_Progress(150)........................:MPID_nem_mpich2_blocking_recv(948)..............:MPID_nem_tcp_connpoll(1720).....................:state_commrdy_handler(1556).....................:MPID_nem_tcp_recv_handler(1446).................: socket closed and i tried to run this fibonnaci program by using MPI 2 library but i have the same error but the question is why it can't spawn more than 8 processes in the same machine or when i run each spawn processes in each machine?

Hi sherif,

Can you show me the code you are using? What distribution are you using? How much memory and how many cores are available on the node(s) you are using? What version of the Intel MPI Library are you using?

I am not able to replicate the behavior you are experiencing, but I am noticing a slowdown at 9 processes per node (24 cores and 24 GB RAM per node on the system I tested). Everything still runs, there is just a pause when starting the 7th process if there are more than 8 on a single node. I will be investigating this further.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools