[openib-general] Re: ANNOUNCE: First usable version of userspace verbs

[Michael S. Tsirkin]

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> I'm happy to announce the initial availability of userspace verbs
> support for brave testers.
> 
> To try this out, check out the roland-uverbs subversion branch:
> 
>     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> 
> and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> userspace verbs support.
> 
> If you want to use a linux-2.6.10 kernel, you will need to apply the
> new linux-2.6.10-backports.diff patch from the branch (which just
> exports get_sb_pseudo()).  No patches at all are required for an
> up-to-date BK or linux-2.6.11-rc4 tree.
> 
> If you use udev, add the rule
> 
>     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your configuration.  Otherwise, create the required device files:
> 
>     mknod /dev/infiniband/uverbs0 c 231 128
>     mknod /dev/infiniband/uverbs1 c 231 129
> 
> and so on for as many HCAs as you have installed.
> 
> The build the userspace libraries in src/userspace/libibverbs and
> src/userspace/libmthca with the usual
> 
>     ./autogen.sh && ./configure && make && sudo make install
> 
> passing whatever parameters to configure you want; you can use
> --prefix to install to another location.  If you set a non-standard
> prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> configure for libmthca.
> 
> Once you have the libraries built and installed, load the ib_mthca and
> ib_uverbs modules. By default, libibverbs will search for driver
> libraries in <prefix>/lib/infiniband; if you installed libmthca
> somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> point to the directory with mthca.so.
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 
>  - Only RC is implemented.  There are not even any functions to call
>    to create UD address handles yet.
>  - Only Tavor mode is supported -- PCI Express HCAs will not work if
>    they are running mem-free firmware.
>  - On x86, only CPUs with SSE will work now.  I'd be surprised if
>    anyone has x86 system with an HCA that doesn't have SSE.
> 
> Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> kernels -- I don't expect any portability problems but I haven't even
> built for other architectures.
> 
> In any case, please give this a spin and let me know how it looks to you.
> 
> My short- and medium-term plans are:
> 
>   1. Catch up on reviewing and applying the patche queue I'm sitting on.
>   2. Land the Arbel mem-free mode support from the roland-uverbs branch
>      onto the main trunk (and merge it upstream once 2.6.11 is out and
>      2.6.12 opens).
>   3. Implement UD support for userspace.  I should have this done before
>      the end of next week.
>   4. Implement mem-free support for userspace.
> 
> Thanks,
>   Roland

Roland, I have implemented a small test for the rdma functionality.
I based it on the pingpong test, the main change being polling on data
instead of completions (but I also changed the clock sampling to use the
realtime clock from -lrt, since it gives a more consistent timing
results on my system).

This is useful as an example of using rdma, and is also useful
as a post send latency benchmark, for tuning (nicer than the send test in that
it let us measure post send separately from poll cq).

Do you want such stuff under libibverbs/examples, or somewhere else?

-- 
MST - Michael S. Tsirkin