Intel MIC with OFED

Intel MIC with OFED



I am trying to build hpc cluster with 4 phi card and 2 nodes.

i am using mlx interconnect for mpi communication.


th cluster details given below:

centos 6.4, intel cluster 2013, MPSS 3.1.2,OFED

Intel Phi card is configured with external network connection on bridge with eth0 interface

opensmd,mpss,openibd,ofed-mic services are running properly.

Only on all 4  intel phi card is used for running service program


 How to identify all 4 phi card communication is running on infiniband not on ethernet.

Please provide solution in detailed description.



3 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Good question. Your bridge is configured the way it is supposed to be. But how do you prove to yourself that the MPI communication is going over the InfiniBand connection even though the bridge points to the ethernet? I will ask around.

In the meantime, if you haven't already read it, you might find section 2.2.9 of the Intel® Xeon Phi™ System Software Developer’s Guide interesting. (You can find the link at You have apparently already read the chapter 2 of the MPSS User's Guide. You might also find Configuring Intel® Xeon PhiTM coprocessors inside a cluster useful. (You can find the link at It is a bit out of date at this point and due for a refresh, but it might still be useful. 



The answers I got:

If you are using Intel(r) MPI Library, you can set the I_MPI_DEBUG environment variable to 3 or higher. The output will tell you what fabric is being used:

$ mpirun –genv I_MPI_DEBUG 3 –n 16 ./exe
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1
[0] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1
[0] MPI startup(): dapl data transfer mode

Otherwise, you can use the InfiniBand tools. 

ibclearcounters                               # clear counters before running a MPI job.
ibdatacounts   < HCA lid #>   <HCA port #>     # check receive data of a specific port
Or ibdatacounters                         # check receive data on all ports

Does this help?

Leave a Comment

Please sign in to add a comment. Not a member? Join today