[openib-general] Re: ANNOUNCE: First usable version of userspace verbs

[Michael S. Tsirkin]

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> I'm happy to announce the initial availability of userspace verbs
> support for brave testers.
> 
> To try this out, check out the roland-uverbs subversion branch:
> 
>     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> 
> and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> userspace verbs support.
> 
> If you want to use a linux-2.6.10 kernel, you will need to apply the
> new linux-2.6.10-backports.diff patch from the branch (which just
> exports get_sb_pseudo()).  No patches at all are required for an
> up-to-date BK or linux-2.6.11-rc4 tree.
> 
> If you use udev, add the rule
> 
>     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your configuration.  Otherwise, create the required device files:
> 
>     mknod /dev/infiniband/uverbs0 c 231 128
>     mknod /dev/infiniband/uverbs1 c 231 129
> 
> and so on for as many HCAs as you have installed.
> 
> The build the userspace libraries in src/userspace/libibverbs and
> src/userspace/libmthca with the usual
> 
>     ./autogen.sh && ./configure && make && sudo make install
> 
> passing whatever parameters to configure you want; you can use
> --prefix to install to another location.  If you set a non-standard
> prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> configure for libmthca.
> 
> Once you have the libraries built and installed, load the ib_mthca and
> ib_uverbs modules. By default, libibverbs will search for driver
> libraries in <prefix>/lib/infiniband; if you installed libmthca
> somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> point to the directory with mthca.so.
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 
>  - Only RC is implemented.  There are not even any functions to call
>    to create UD address handles yet.
>  - Only Tavor mode is supported -- PCI Express HCAs will not work if
>    they are running mem-free firmware.
>  - On x86, only CPUs with SSE will work now.  I'd be surprised if
>    anyone has x86 system with an HCA that doesn't have SSE.
> 
> Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> kernels -- I don't expect any portability problems but I haven't even
> built for other architectures.
> 
> In any case, please give this a spin and let me know how it looks to you.

The pingpong test fails for me, on x86_64. (The executable being a 64
bit one). My kernel is 2.6.10.

So far, I traced it down to ibv_cmd_create_qp in src/cmd.c, where this command
fails:

183             if (write(pd->context->cmd_fd, cmd, cmd_size) != cmd_size)

By adding some prints I see

cmd 0xbfffef30 returned 0xffffffff expected 0x48, error=22

so a write failed with errno 22 which is EINVAL.

gdb backtrace:

#0  ibv_cmd_create_qp (pd=Variable "pd" is not available.) at src/cmd.c:190
#1  0x0000002a95cb7add in mthca_create_qp (pd=Variable "pd" is not available.
) at src/verbs.c:225
#2  0x0000002a9566ed7a in ibv_create_qp (pd=Variable "pd" is not available.
) at src/verbs.c:117
#3  0x0000000000401fcf in pp_init_ctx (ib_dev=0x509850, size=4096,
    rx_depth=500, port=1) at examples/pingpong.c:308
#4  0x00000000004027de in main (argc=1, argv=0x7fbffff378)
    at examples/pingpong.c:550


-- 
MST - Michael S. Tsirkin