[openib-general] Re: unable to run ibv_uc_pingpong and other example binaries

[Michael S. Tsirkin]

Quoting r. Joe Damato <damato at psc.edu>:
> Subject: unable to run ibv_uc_pingpong and other example binaries
> 
> Hello,
>   
>    First I would like to apologize for the long email.
>   
>    I am currently using the gen2 revision 2734 openib source and kernel 
> 2.6.11. I have compiled and installed libibverbs, libmthca and libibcm. 
> I have also created the /dev/infiniband devices, and I have my card 
> flashed to the latest firmware version.
> 
> [root at frodo examples]# pwd
> /cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
> [root at frodo examples]# ./ibv_uc_pingpong

You want to run .libs/ibv_uc_pingpong I think.
The one you are running is a shell script.

> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found
> [root at frodo examples]# ./ibv_rc_pingpong
> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found

Hmm. I didnt know libibverbs depends on libucm. 
Can you strace it?

> but....
> 
> [root at frodo examples]# cd /dev/
> [root at frodo dev]# ls infiniband/
> issm0  issm1  ucm  umad0  umad1  uverbs0  uverbs1
> [root at frodo dev]# ls -al infiniband/
> total 0
> drw-rw-rw-  2 root root      180 Jul 19 11:36 .
> drwxr-xr-x  8 root root    13940 Jul 19 11:35 ..
> crw-rw-rw-  1 root root 231,  64 Jul 19 11:35 issm0
> crw-rw-rw-  1 root root 231,  65 Jul 19 11:35 issm1
> crw-rw-rw-  1 root root 231, 255 Jul 19 11:36 ucm
> crw-rw-rw-  1 root root 231,   0 Jul 19 11:35 umad0
> crw-rw-rw-  1 root root 231,   1 Jul 19 11:35 umad1
> crw-rw-rw-  1 root root 231, 192 Jul 19 11:36 uverbs0
> crw-rw-rw-  1 root root 231, 193 Jul 19 11:36 uverbs1

BTW, you probably dont want umad and issm world-accessible, it can be
a security hole: these are management interfaces.

> [...]
> I currently have 8 other machines with IB, however when trying to 
> run ibv_rc_pingpong they all report the same error  (each of the 8 
> systems are running the same kernel and all cards are flashed to the 
> latest firmware):
> 
> ost0# ./ibv_rc_pingpong
> Couldn't get local LID
> 
> but when checking ibstat it shows that Port 2 is active

I think these tests assume that since local LID is 0 sm isnt running.
Try running opensm on one of the nodes.

-- 
MST