Installed MPSS with ofed support, not able to get access to xeon phi's over ib

Installed MPSS with ofed support, not able to get access to xeon phi's over ib

I installed mpss 3.3 with ofed on my 5110p based server. IB is mlx_4. Node IB is working fine. I am able to login to a remote xeon phi using external bridging support. I am running these services: mpss, openibd, ofed-mic, and mpxyd. I am still not able to access xeon phi over IB (not even on same node).

I have configured ipoib.conf to set the ip of xeon phi to be on the same network as my other nodes. What could be the problem?
No problem connecting xeon phis within single node using scif. Some how i am not able to utilize ccl.


System details:

OS: RHEL 6.3
HCA Driver: mlx_4
MPSS: 3.3 gold update
Xeon phi ROM: updated
Hardware: Supermicro with intel xeon e5 processor with Mellanox HBA

3 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

I don't have any real experience using IPoIB, although I might be about to gain some. But for starters, can you use InfiniBand verbs at all to access the card? Does ibv_devinfo look right? (You can see an example output at I guess what I am trying to ask, is it just IPoIB or are the problems more basic than that?

  1. Are you sure that you are using "MPSS 3.3 gold update"? MPSS 3.3 is indeed the latest version, but the last public "gold update" was in MPSS 2.1. If you are have a "gold update" version, you may be off by one paragraph in the download page. Here is the correct download location:
  2. I recently configured IPoIB for Xeon Phi with MPSS 3.x, and the stumbling block for me were firewalls. 
  3. Here are the contents of /etc/mpss/ipoib.conf that worked for my network:
mic0_ib0=" netmask"
mic1_ib0=" netmask"
mic2_ib0=" netmask"
mic3_ib0=" netmask"

Also, I noticed that a visual indicator that service ofed-mic is happy with your /etc/mpss/ipoib.conf is output that contains "ib0" after "mic*" like below. If it does not look like that, something is off in ipoib.conf:

[root@c001-n001 ~]# service ofed-mic start
Starting OFED Stack:
host                                                       [  OK  ]
mic0 - ib0 - mic1 - ib0 - mic2 - ib0 - mic3 - ib0 -        [  OK  ]

Maybe this will help.

Leave a Comment

Please sign in to add a comment. Not a member? Join today