By some reason, kernel modules, built from intel-mic-ofed-kmod source package, don't want to load on CentOS 6.3, for example:
[root@node001 ~]# modprobe ib_umad
FATAL: Error inserting ib_umad (/lib/modules/2.6.32-279.22.1.el6.x86_64/updates/drivers/infiniband/core/ib_umad.ko): Unknown symbol in module, or unknown parameter (see dmesg)
It seems CentOS 6.3 and intel-mic-ofed-kmod sources are not compatible (or I am doing something wrong). We are using the latest publicly available MPSS stack (Update 1) and we build intel-mic-ofed-* packages on boot. So, could you please answer (or point me to the correct documentation) the following two questions:
- There are at least 3 widly used OFED versions: OFA OFED, Mellanox OFED and QLogic OFED. What exact MPSS versions work with what OFED versions on what linux distrubutions? I suspect the correct answer changes all the time quickly, but I will be highly approtiated if somebody provides us with at least the current state.
- As far as I understand, the main reason to set up OFED on a host is to emulate HCA and allow ib-wise applications to communicate between the host and the card via "infiniband" (using rdma). Is it possible to use several MICs (installed on *different* hosts phisicaly connected to ib-switch) to get all advantages of ib communication between them? For example, it would be nice to run a mvapich2 native application on several MICs (on different hosts) in a cluster using ib only (Probably HCA emulation makes ib communication slower, but I am not sure).