[openib-general] Start of an IPoIB FAQ

[Hal Rosenstock]

Hi,

The start of an IPoIB FAQ may be in order.

Something along the lines of:

ping doesn't work between IPoIB nodes. What should I do ?

First, verify that the ports are active. This can be done via:

cat /sys/class/infiniband/mthca0/ports/1/state

This should indicate 4: ACTIVE

assuming the HCA is mthca0 and port 1 is plugged in.

Next, verify the firmware version via

cat /sys/class/infiniband/mthca0/fw_ver

For PCI-X HCAs, version 3.2.0 is recommended. For PCIe HCAs, version 
4.5.3 is recommended.

If these versions of the firmware are being used, indicate the
configuration and which SM is being utilized.

Do /sys/class/net/ib0/statistics/rx_packets and/or "tcpdump -i ib0"
show anything on the other nodes when you try to ping or something?

There are 2 levels of IPoIB debug which can be enabled when building:
IP-over-InfiniBand debugging and IP-over-InfiniBand data path debugging.
The latter has performance implications and should only be enabled when
all else fails. Enable the first level of IPoIB debug and then:

mount -t ipoib_debugfs none /ipoib_debufs/
cat /ipoib_debugfs/ib0_mcg

-- Hal