Working with Mellanox* InfiniBand Adapter on System with Intel® Xeon Phi™ Coprocessors

InfiniBand* is a network communications protocol commonly used in the HPC area because the protocol offers very high throughput. Intel and Mellanox* are among the most popular InfiniBand* adapter manufacturers. In this blog, I will share my experience of installing and testing Mellanox* InfiniBand* adapter cards with three different versions of OFED* (Open Fabrics Enterprise Distribution), OpenFabrics OFED-, OpenFabrics OFED-3.5.2-mic and Mellanox* OFED 2.1, on systems containing Intel® Xeon Phi™ coprocessors.

在采用英特尔® 至强融核™ 协处理器的系统上配置 Mellanox* InfiniBand 适配器

InfiniBand* 是 HPC 区域中经常使用的网络通信协议,因为该协议可以提供非常高的吞吐量。 英特尔和 Mellanox* 是最受欢迎的 InfiniBand* 适配器生产商。 在本博客中,我将介绍如何使用三个版本的 OFED* (Open Fabrics Enterprise Distribution) — OpenFabrics OFED-、OpenFabrics OFED-3.5.2-mic 和 Mellanox* OFED 2.1 在采用英特尔® 至强融核™ 协处理器的系统上安装和测试 Mellanox* InfiniBand* 适配器卡。

为了允许协处理器上的原生应用与 Mellanox* InfiniBand 适配器进行通信,必须启用协处理器通信链路 (CCL)。 当使用 Mellanox* InfiniBand 适配器时,上面提到的三种 OFED 堆栈支持 CCL。

1. 硬件安装

使用两种系统,每种系统配备一台英特尔® 至强™ E5-2670 2.60 GHz 处理器和两台英特尔® 至强融核™ 协处理器。 两个系统都运行 RHEL 6.3。 它们使用千兆以太网适配器并通过千兆以太网路由器进行连接。

Troubleshooting OFED* for Use with the Intel® MPSS

This is a list of what are hopefully helpful hints for troubleshooting problems with OFED* and InfiniBand* when used with the Intel® Xeon Phi™ coprocessor.

1) Finding Documentation

The primary sources for information on OFED for the Intel Xeon Phi coprocessor are:

Incompatibility between the Intel® MPI Library and certain versions of the Mellanox* OFED drivers


There is an incompatibility between the Intel® MPI Library and some versions of the Mellanox* OFED drivers when using the DAPL* fabric.  This has been reported with Mellanox* OFED 2.1-1.0.0, and could exist in other versions.


You will likely see Segmentation Faults in MPI_Init when running.  A stack trace could show:

  • Linux*
  • Server
  • Intel® MPI Library
  • incompatibility
  • seg fault
  • segmentation fault
  • linux drivers
  • InfiniBand
  • Message Passing Interface
  • Cluster-Computing
  • Enabling Connectionless DAPL UD in the Intel® MPI Library

    What is DAPL UD?

    Traditional InfiniBand* support involves MPI message transfer over the Reliable Connection (RC) protocol. While RC is long-standing and rich in functionality, it does have certain drawbacks: since it requires that each pair of processes setup a one-to-one connection at the start of the execution, memory consumption could (at the worst case) grow linearly as more MPI ranks are added and the number of pair connections grows.

  • Entwickler
  • Linux*
  • Microsoft Windows* (XP, Vista, 7)
  • Server
  • C/C++
  • Fortran
  • Fortgeschrittene
  • Intel® MPI Library
  • user datagrams
  • ud
  • dapl ud
  • IB
  • InfiniBand
  • scalability
  • Message Passing Interface
  • Cluster-Computing
  • InfiniBand abonnieren