Lustre on Xeon Phi

Lustre on Xeon Phi

Do folks have instructions of howto install/run Lustre on Xeon Phi (as client) ? I'm on a relatively old MPSS  (mpss_gold_update_3, mic-2.1.6720). My server runs 1.8.8 version of Lustre.

Thanks, Wendy

7 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Have you checked out:

With the next release of the MPSS, some of the information in here will be out of date (in particular, file locations and some configuration information) but for the version you are using, this document should help. There is a section on compiling the Lustre software and in the section discussing the boot script he wrote, Michael goes into starting the Lustre mods.

Frances, Thanks for the pointer. I did check out the document but building a GNU build environment inside Xeon Phi is too involved for my purpose. With the help from Dmitry Eremin (, the Lustre client is now up and running by doing:

  1. Download the SOURCE package from
  2. Unpack it and find the package-full_src-k1om.tar.bz2 file (in gpl directory). Unpack (again) this archive at /tmp/src. At /tmp/src,  execute the "make defconfig-miclinux", followed by "make -C card/kernel ARCH=k1om modules_prepare"
  3. Make sure "intel-mic-gpl-<version>.x86_64" package installed.
  4. export PATH=/usr/linux-k1om-4.7/bin:$PATH
  5. Obtain luster source (git clone git://
  6. Goto top of lustre tree (lustre-release) : do a "sh", followed by "./configure --with-linux=/tmp/src/card/kernel --disable-server --disable-tests --disable-doc --without-o2ib --host=x86_64-k1om-linux --build=x86_64-pc-linux", then "make rpms"
  7. Look for lustre-client-mic RPMs in the usual rpmbuild directory (in RHEL case, it is /root/rpmbuild) , install them. 
  8. Restart mpss
  9. All set to go :)

-- Wendy

Thanks for posting the directions. Glad it is working.


Hello everyone,

We just gave a try with mpss 3.2 and we faced some trouble to build lustre client for the phi, starting with missing or renamed packages from previous mpss releases - well, we moved from 2.x to 3.2 straight, kind of a big jump...

Any update/clues/advices will be more than welcome !

Thanks in advance,


Hi Pierre, 

here I adapted some instructions I saw from a colleague, but have not verified in full.   I am hoping these will contain enough information to help unblock you -- please let us know if this worked


  • Only Lustre* version 2.5 and above of the Lustre client can actually be cross-compiled with MPSS 3.2 and OFED
  • It is required to have OFED version Infiniband drivers installed

Steps for installation:

1. Download the SOURCE mpss-3.x-k1om.tar and OS specific files from the MPSS download page

2. Unpack from them "kernel-dev-*.rpm", "ofed-driver-*-devel-*.rpm" and "linux-*.tar.bz2" files

3. Prepare Intel MPSS sources for Lustre build:

  •    rpm2cpio kernel-dev-*.rpm | cpio -idm
  •    rpm2cpio ofed-driver-*-devel-*.rpm | cpio -idm
  •    tar xjvf linux-*.tar.bz2 && cd linux-*
  •    cp -f ../boot/config-* .config
  •    make ARCH=k1om silentoldconfig modules_prepare
  •    cp -f ../boot/Module.symvers-* Module.symvers

4. Now build Lustre:

  • ./opt/mpss/3.x/environment-setup-k1om-mpss-linux
  • sh
  • ./configure --with-linux=<path>/linux-2.6.38+mpss3.x --with-o2ib=<path>/usr/src/ofed-driver-*.el6.x86_64 --host=k1om-mpss-linux --build=x86_64-pc-linux
  • make rpms

Only two Lustre* RPMs should be installed on the host:

  • lustre-client-mic-<version>.x86_64.rpm
  • lustre-client-mic-modules-<version>.x86_64.prm

Additional step:    ssh mic? "echo 'options Inet networks=\"o2ib0(ib0)\"' > /etc/modprobe.d/lustre.conf"

The Host configuration in /etc/modprobe.d/lustre.conf should look like this:   options Inet networks="o2ib0(ib0)" and mounting the Lustre filesystem should be something like this:   mount -t lustre

The coprocessor's /etc/modprobe.d/lustre.conf should have the options Inet networks="o2ib0(ib0)" entry and the mount command will look like this:  mount.lustre


Hi Team,


As per the above mention steps we are able to build the lustre mic module with o2ib.

But at the time of mounting lustre file system we are getting at lustre server side

Aug  5 23:57:54 IO1 kernel: Lustre: Skipped 1 previous similar message
Aug  5 23:59:32 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept on (ib1:1: bad dst nid
Aug  5 23:59:57 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept on (ib1:1: bad dst nid
Aug  6 00:00:22 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept on (ib1:1: bad dst nid
Aug  6 00:11:01 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) REJECTED 28
Aug  6 00:11:26 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) REJECTED 28


MIC details are given below:-

[root@phi3 ~]# ibv_devices
    device                 node GUID
    ------              ----------------
    scif0               4c79bafffe340459
    mlx4_0              f452140300831c60
[root@phi3 ~]#

[root@phi3 ~]# micinfo
MicInfo Utility Log
Copyright 2011-2013 Intel Corporation All Rights Reserved.

Created Wed Aug  6 00:36:48 2014

        System Info
                HOST OS                 : Linux
                OS Version              : 2.6.32-431.el6.x86_64
                Driver Version          : 3.3-1
                MPSS Version            : 3.3
                Host Physical Memory    : 65920 MB

Device No: 0, Device Name: mic0

                Flash Version            :
                SMC Firmware Version     : 1.14.4616
                SMC Boot Loader Version  : 1.8.4326
                uOS Version              :
                Device Serial Number     : ADKC32600556


please guide to resolve the problem


THank YOu

Atul Yadav



Leave a Comment

Please sign in to add a comment. Not a member? Join today