[ofa-general] Installing openIB on Linux FC5

[Hossein Pourreza]

Hi all,

I am new to infiniband stuff and am trying to configure an infiniband-based
cluster using Linux FC 5. I downloaded the OFED-1.0 and tried to install it on
cluster nodes. Now I can load the kernel modules without any error but I cannot
run a simple test like ibv_ud_pingpong to check the connectivity of nodes in
user-level.

I loaded the following devices:

ib_umad                25713  0 
ib_ucm                 26569  0 
ib_cm                  42521  1 ib_ucm
ib_uverbs              47889  1 ib_ucm
ib_mthca              133445  0 
ib_ipoib               61361  0 
ib_sa                  25341  2 ib_cm,ib_ipoib
ib_mad                 46969  4 ib_umad,ib_cm,ib_mthca,ib_sa
ib_core                63809  8 ib_umad,ib_ucm,ib_cm,ib_uverbs,ib_mthca,ib_ipoib,ib_sa,ib_mad

Also I have the following devices in /dev/infiniband

crw-rw---- 1 root root 231,  64 Jun  4 14:54 issm0
crw-rw---- 1 root root 231,  65 Jun  4 14:54 issm1
crw-rw---- 1 root root 231, 224 Jun  4 14:34 ucm0
crw-rw---- 1 root root 231,   0 Jun  4 14:54 umad0
crw-rw---- 1 root root 231,   1 Jun  4 14:54 umad1
crw-rw-rw- 1 root root 231, 192 Jun  4 14:34 uverbs0

ibroute shows all nodes and the switch and everything looks fine.
When I run ibv_ud_pingpong on the two nodes I am getting the following messages:

node 1 (server):
local address:  LID 0x0002, QPN 0x150406, PSN 0xb3a00d
remote address: LID 0x0003, QPN 0x0c0406, PSN 0x8f0f99


node 2 (client):
local address:  LID 0x0003, QPN 0x0c0406, PSN 0x8f0f99
remote address: LID 0x0002, QPN 0x150406, PSN 0xb3a00d

There is no message after these two lines. I am wondering if they are sending
any packets or not. I should say that although I have given ip addresses to
infiniband cards (ib0) they cannot ping each other using the normal Linux ping
tool.

Here is the result of ifconfig on these nodes:

node 1 (server)

ib0       Link encap:InfiniBand  HWaddr 00:00:04:04:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:172.16.28.61  Bcast:172.16.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:128 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

node 2 (client):
ib0       Link encap:InfiniBand  HWaddr 00:00:04:04:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:172.16.28.62  Bcast:172.16.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:128 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Any help will be greatly appreciated.

Hossein

-- 
Hossein Pourreza		 			mail:<pourreza at cs.umanitoba.ca>    
Department of Computer Science		URL: http://www.cs.umanitoba.ca/~pourreza
University of Manitoba  			Phone: 204-488-5611            
Winnipeg, Manitoba, Canada R3T 2N2