Infiniband Troubleshooting

 

This comes from a stack of notes i compiled when faced with issues on an infiniband deployment that needed troubleshooting.

1)    InfiniBand Hardware Checks

 Carry out the following hardware checks:

a)      Make sure that the InfiniBand switches are powered up.

b)      Make sure that the each machine is powered up.

c)      Make sure that each machine is fitted with a functional InfiniBand adapter. If the adapter is not functional insert a new one.

d)      Check that the InfiniBand adapter is supported for the architecture of the cluster.

e)      Once the card is fitted into the server, run the command below to check that it is visible:

 lspci | grep InfiniBand

03:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR,

PCIe 2.0 5GT/s] (rev a0)

 If the card is not recognized  replace the faulty card by a card that is known to work properly, and run the above command again.

f)       Once the card is fitted into the server, make sure that it is connected properly by checking that the cable connectors are fully inserted into the ports.

• • Green LEDs, when lit, indicate that the physical link is up. Yellow LEDs, when lit, indicate that the logical link is up, and their blinking rate indicates link activity.
  • If a green LED is not lit, swap the cable with a cable that is known to work properly and test again. If it still fails, try the cable on another switch port. If it works, then the former switch port may be out of order or disabled by the system administrator.

2)    InfiniBand Software Installation Check

Make sure that following packages are installed:

1)      kernel-ib

2)      kernel-ib-scripts

3)      infiniband-diags

4)      libibmad

5)      libibumad

6)      opensm-libs

Use the command to check that a given package is installed: rpm -q <package name>

Depending on the interconnect architecture, the node may also be used as an OpenSM subnet manager. In this case, the opensm package should also be installed.

3)    Check the openibd service

 The openibd service is a script that loads all the InfiniBand drivers listed in the openibd configuration file.

 This configuration file is located in the /etc/infiniband directory, and its generic name is openib_<kernel release>.conf. The example below details the openib settings for a 2.6.18-128.el5.Bull.2 kernel configuration file. cat /etc/infiniband/openib_2.6.18-128.el5.Bull.2.conf

4)    Modules and Drivers for InfiniBand networks

 A range of modules and drivers are possible for InfiniBand networks, and include the following:

a)      Core modules

• ib_addr : InfiniBand address translation

• ib_core : Core kernel InfiniBand API

b)      Hardware support

 • mlx4_ib Mellanox ConnectX HCA Infiniband driver

• mlx4_core Mellanox ConnectX HCA low-level driver

• ib_mthca Mellanox Infiniband HCA low-level driver

• qlgc_vnic QLogic virtual NIC (VNIC) driver

• mlx4_en Mellanox ConnectX HCA Ethernet driver

c)      IP over IB modules

• ib_ipoib IP-over-InfiniBand net driver

• ipoib_helper Container for ipoib neighbour destructor

d)      Subnet manager related modules

• ib_mad IB MAD API

• ib_sa Subnet administration query support

• ib_umad Userspace MAD packet access

e)      MPI related modules

 • ib_uverbs Userspace verbs access

 f)       Connection managers

• ib_ucm Userspace connection manager access

• ib_cm Connection manager

• iw_cm iWARP connection manager

• rdma_cm Generic RDMA connection manager

• rdma_ucm RDMA userspace connection manager

g)     Socket direct protocol

 • ib_sdp Socket direct protocol (SDP).

 h)      Storage

• ib_iser iSER (iSCSI Extensions for RDMA) datamover

• ib_srp InfiniBand SCSI RDMA Protocol initiator

• ib_srpt InfiniBand SCSI RDMA Protocol target

• iscsi_tcp iSCSI/TCP data-path

• Libiscsi iSCSI library functions

• scsi_transport_iscsi iSCSI transport interface

5)    Starting the openibd service

 a.      Start the openibd service using the command: # service openibd start

 This will give output similar to that below:

 Loading HCA driver and Access Layer: [ OK ]

Setting up InfiniBand network interfaces:

Bringing up interface ib0: [ OK ]

Bringing up interface ib1: [ OK ]

Setting up service network . . . [ done ]

 If openibd service fails to start, check that the configuration file is correct.

 Also, please make sure that the service is automatically started when the node boots: chkconfig –list openibd

 b.      Run the command below to ensure that the openibd service starts automatically: # chkconfig openibd on

If the openibd service fails to start, make sure that its version matches the version of the kernel that is running: rpm -q kernel-ib

 

6)    Check the InfiniBand Adapters

 Mellanox Firmware Issues

 Check that the InfiniBand adapters’ firmware is up-to-date.

Check the InfiniBand firmware version, by using the command: mthca_fw_upd –t : mlx4_0: Firmware 2.6.0 is up to date

Alternatively, the current firmware version can be seen by running the command: cat /sys/class/infiniband/*/fw_ver

7)    Physical States of the port

Once the drivers listed in the openib_<kernel-release>.conf file have been loaded by the openibd service successfully, the physical state of the ports should be checked. InfiniBand adapters may have several ports and depending on the cluster topology, all the ports may not be connected to a switch.

 Use the ibstat command as follows to check the physical state of the adapter port(s):

 ibstatus

CA ‘mlx4_0’ CA type: MT26428

Number of ports: 2

Firmware version: 2.6.0

Hardware version: a0

Node GUID: 0x0002c903000262da

System image GUID: 0x0002c903000262dd

Port 1:

State: Active

Physical state: LinkUp

Rate: 40

Base lid: 1

LMC: 0

SM lid: 1

Capability mask: 0x00000000

Port GUID: 0x0002c903000262db

Port 2:

State: Down

Physical state: Polling

Rate: 10

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x00000000

Port GUID: 0x0002c903000262dc

 In the example above, we can see that the first port is physically connected (LinkUp) and that a Subnet Manager is present in the InfiniBand network. The second port is not physically connected (Polling) and as a result cannot see a Subnet Manager (State: Down).

 The following physical states are possible:

a)     Sleep

b)     Polling

   i.            No cable is connected to the adapter port: try to reconnect the cable and check the LED status.

  ii.            The cable is broken and needs to be replaced.

  iii.           The port on the other end of the cable is disabled.

 c)      Disabled : Enable the port using the ibportstate command

d)      PortConfigurationTraining : Wait for few seconds until the port changes state. If it does not change, check if the adapter and the switch that it is connected to are compatible.

e)      LinkUp : The port is physically connected.

f)       LinkErrorRecovery : The cable is faulty. Replace the cable. Alternatively, the ibstatus command can be used to check the adapter status.

8)    Check the OpenSM Subnet Manager

Once the physical state for the port is set to LinkUp, the Subnet Manager needs to be started for the InfiniBand network to function. In the example below, the physical state is LinkUp, but no Subnet Manager is running (State: Initializing):

ibstat

Port 1:

State: Initializing

Physical state: LinkUp

The absence of the Subnet Manager is confirmed by running the sminfo command:  sminfo: iberror: failed: query

Two scenarios are possible here:

• The InfiniBand network includes Voltaire or Mellanox managed switches. Refer to the switch documentation to fix this.

• The InfiniBand network does not include managed switches. In this case, OpenSM should be installed on one or more nodes within the cluster.

9)    Check the Switch Firmware

 The information in this section applies to non-managed switches only. Refer to the switch documentation if the switch is managed internally. The following specific packages are required in order to check the(all) switch(s) firmware version.

• infiniband-diags

• mft

• ibsw_fw_update

10)    Check the IP over IB interfaces

Each InfiniBand port on the node should have a corresponding entry in /etc/modprobe.conf configuration file. The entry describes the IP interface name and the corresponding kernel module to use. Example for 2 interfaces named ib0 and ib1:

alias ib0 ib_ipoib

alias ib1 ib_ipoib

 After adding the appropriate lines to /etc/modprobe.conf configuration file, run the following command: # depmod –a

The IPoIB interface is configured by a file in the /etc/sysconfig/network-script directory which is generated using the config_ip command.