Assign specific mpi tasks to specific IB interfaces

Assign specific mpi tasks to specific IB interfaces

I have a system with 16 cores and 2 IB interfaces. I need to use both rails, so have been using export I_MPI_OFA_NUM_ADAPTERS=2 export I_MPI_OFA_NUM_PORTS=1 export I_MPI_OFA_RAIL_SCHEDULER=ROUND_ROBIN export I_MPI_FABRICS="shm:ofa"But this has the undesirable effect of sending some of the data from the second 8 cores to the IB on the first eight cores and vice-versa.I triedexport I_MPI_OFA_NUM_ADAPTERS=2export I_MPI_OFA_NUM_PORTS=1export I_MPI_OFA_RAIL_SCHEDULER=PROCESS_BINDexport I_MPI_FABRICS="shm:ofa"But this assigns process 0 to interface 1, process 1 to interface 2, process 3 to interface 1, etc. This sends alot of data from one set of cores to the opposing interface. Furthermore, the code is written so that it assumes that process 0 and process 1 are in the same cpu so it tries to do a cyclic data movement.I need to assign process 0 - 7 to interface 1 and process 8 - 15 to interface 2.Is this possible?ThanksDavidThanks

3 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.
James Tullos (Intel)'s picture

Hi David,

Try using something like the following:

export I_MPI_FABRICS=shm:ofa

mpirun -n 8 -env I_MPI_OFA_ADAPTER_NAME adap1  : -n 8 -env I_MPI_OFA_ADAPTER_NAME adap2 

This should set the first 8 processes to use the first adapter, and the next 8 to use the second adapter.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

Hi David,

I need to make some clarifications.
If you use I_MPI_FABRICS=shm:ofa it means that 'shm' will be used for INTRA-node communication and 'ofa' will be used for INTER-node communication.
Since you are going to use OFA for intra-node communication you need to set I_MPI_FABRICS to 'ofa' or 'ofa:ofa'.
And all other paratemers as James mentioned.
Please give it a try and compare results with default settings. It would be nice if you could share with us results of different runs.

Regards!
Dmitry

Login to leave a comment.