[openib-general] Re: ANNOUNCE: First usable version of userspace verbs

[Michael S. Tsirkin]

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
>     Michael> The pingpong test fails for me, on x86_64. (The
>     Michael> executable being a 64 bit one). My kernel is 2.6.10.
> 
> OK, I installed x86_64 userspace and got user verbs working.  The
> first problem you ran into was because on a 32-bit arch,
> 
>     struct {
>             uint64_t a;
>             uint32_t b;
>     };
> 
> has size 12 but on a 64-bit arch it has size 16.  Because of the way I
> tested sizes of parameters passed into the kernel, it was basically
> looking for userspace to pass a size of 12.  I fixed this by padding
> all the structs to a multiple of 8 bytes so we don't run into this.
> 
> I also fixed a few warnings and other small glitches, and now the
> pingpong test works for me from both 32- and 64-bit userspace.
> 
> Since the ABI is different, you'll have to rebuild both the kernel
> modules and the userspace libraries when updating.
> 
> Thanks,
>   Roland
> 

Hmm.
# /usr/local/bin/ibv_pingpong
(no diagnostic is produced)
# /usr/local/bin/ibv_pingpong -d uverbs0
IB device uverbs0 not found

swlab155:/usr/src/roland-uverbs/src/userspace/libibverbs # ls -l
/dev/infiniband/
total 124
drwxr-xr-x   2 smtp root     4096 Feb 24 15:31 .
drwxr-xr-x  36 smtp root   118784 Feb 27 12:53 ..
crw-------   1 smtp root 231,  64 Feb 24 15:31 issm0
crw-------   1 smtp root 231,  65 Feb 24 15:31 issm1
crw-------   1 smtp root 231,   0 Feb  3 19:09 umad0
crw-------   1 smtp root 231,   1 Feb  3 19:09 umad1
crw-rw-rw-   1 smtp root 231, 128 Feb 24 15:31 uverbs0

Shall uverbs and umad share the same major?

swlab155:/usr/src/roland-uverbs/src/userspace/libibverbs # cat
/proc/devices | grep 231
231 infiniband_mad
231 infiniband_verbs

somewhat surprising.

Nodes are created by udev

# tail /etc/udev/udev.rules
KERNEL="mouse*",        NAME="input/%k"
KERNEL="event*",        NAME="input/%k"
KERNEL="js*",           NAME="input/%k"
KERNEL="ts*",           NAME="input/%k"

KERNEL="umad*", NAME="infiniband/%k"
KERNEL="issm*", NAME="infiniband/%k"
KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"

Interestingly, if I remove the node:

# rm /dev/infiniband/uverbs0
# /usr/local/bin/ibv_pingpong
Couldn't get context for mthca0

shouldnt it rather report that uverbs0 is missing?

-- 
MST - Michael S. Tsirkin