Troubleshooting InfiniBand connection issues using OFED tools

The Open Fabrics Enterprise Distribution (OFED) package has many debugging tools available as part of the standard release. This article describes the use of those tools to troubleshoot the hardware and firmware of an InfiniBand fabric deployment.

First, the /sys/class sub-system should be checked to verify that the hardware is up and connected to the InfiniBand fabric.  The following command will show the InfiniBand hardware modules recognized by the system:

ls /sys/class/infiniband

This example will use the module mlx4_0, which is typical for Mellanox ConnectX* series of adapters. If this, or a similar module, is not found, refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Next, check the state of the InfiniBand port:

cat /sys/class/infiniband/mlx4_0/ports/1/state

This command should return “ACTIVE” if the hardware is initialized, and the subnet manager has found the port and added the port to the InfiniBand fabric. If this command returns “INIT” the hardware is initialized, but the subnet manager has not added the port to the fabric yet. 

If necessary, start the subnet manager

/etc/init.d/opensmd start

Once the port on the head node is in the “ACTIVE” state, check the state of the InfiniBand port on all the compute nodes to ensure that all of the Infiniband hardware on the compute nodes has been initialized, and the subnet manager has added all of the compute nodes ports on to the fabric. This article will use the pdsh tool to run the command on all nodes:

pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/state

All nodes should report “ACTIVE”. If a node reports it cannot find the file, ensure the OpenIB drivers is loaded on that node. Refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Once all of the compute nodes report that port 1 is “ACTIVE”, verify the speed on each port using the following commands: 

cat /sys/class/infiniband/mlx4_0/ports/1/rate
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/rate

This is a good first check for a bad cable or connection.  Each port should report the same speed. For example, the output for double data rate (DDR) InfiniBand cards will be similar to “20 Gb/sec (4X DDR)”.

Once the above basic checks are complete, more in-depth troubleshooting can be performed. The main OFED tool for troubleshooting performance and connection problems is ibdiagnet. This tool runs multiple tests, as specified on the command line during the run, to detect errors related to the subnet, bad packets, and bad states. These errors are some of the more common seen during initial setup of Infiniband fabrics. 

Run ibdiagnet with the following command line options:

ibdiagnet –pc –c 1000

The output will be similar to this:

Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib64/ibdm1.2
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering ... 17 nodes (1 Switches & 16 CA-s) discovered.

-I---------------------------------------------------
-I- Bad Guids/LIDs Info
-I---------------------------------------------------
-I- No bad Guids were found

-I---------------------------------------------------
-I- Links With Logical State = INIT
-I---------------------------------------------------
-I- No bad Links (with logical state = INIT) were found

-I---------------------------------------------------
-I- PM Counters Info
-I---------------------------------------------------
-I- No illegal PM counters values were found

-I---------------------------------------------------
-I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)
-I---------------------------------------------------

-I---------------------------------------------------
-I- IPoIB Subnets Check
-I---------------------------------------------------
-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
-W- No members found for group

-I---------------------------------------------------
-I- Bad Links Info
-I- Errors have occurred on the following links
(for errors details, look in log file /tmp/ibdiagnet.log):
-I---------------------------------------------------
Link at the end of direct route "1,5"
----------------------------------------------------------------
-I- Stages Status Report:
STAGE Errors Warnings
Bad GUIDs/LIDs Check 0 0
Link State Active Check 0 0
Performance Counters Report 0 0
Partitions Check 0 0
IPoIB Subnets Check 0 1
Link Errors Check 0 0

Please see /tmp/ibdiagnet.log for complete log
----------------------------------------------------------------
-I- Done. Run time was 9 seconds.

The warning “No members found for group” can safely be ignored.

In this example, a bad link was found: “Link at the end of direct route “1,5”.”  "1,5" refers to the LID numbers associated with the individual ports. The following commands can be used to identify the LID numbers associated with each port:

cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –a /sys/class/infiniband/mlx4_0/ports/1/lid

This command generates a list of LIDs associated with nodes. In the output of the above command, locate the entries for 0x1 and 0x5.  0x1 is likely the head node.   For errors of this type, reseat or replace the InfiniBand cable connecting the node corresponding to LID 0x5.

Finally, run ibdiagnet once more time to verify there are no errors, and then to check the error state of each port. Each test should pass.

ibdiagnet –pc –c 1000
ibcheckerrors.
For more complete information about compiler optimizations, see our Optimization Notice.