[openib-general] ANNOUNCE: First usable version of userspace verbs

[Roland Dreier]

I'm happy to announce the initial availability of userspace verbs
support for brave testers.

To try this out, check out the roland-uverbs subversion branch:

    svn co https://openib.org/svn/gen2/branches/roland-uverbs

and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
userspace verbs support.

If you want to use a linux-2.6.10 kernel, you will need to apply the
new linux-2.6.10-backports.diff patch from the branch (which just
exports get_sb_pseudo()).  No patches at all are required for an
up-to-date BK or linux-2.6.11-rc4 tree.

If you use udev, add the rule

    KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"

to your configuration.  Otherwise, create the required device files:

    mknod /dev/infiniband/uverbs0 c 231 128
    mknod /dev/infiniband/uverbs1 c 231 129

and so on for as many HCAs as you have installed.

The build the userspace libraries in src/userspace/libibverbs and
src/userspace/libmthca with the usual

    ./autogen.sh && ./configure && make && sudo make install

passing whatever parameters to configure you want; you can use
--prefix to install to another location.  If you set a non-standard
prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
configure for libmthca.

Once you have the libraries built and installed, load the ib_mthca and
ib_uverbs modules. By default, libibverbs will search for driver
libraries in <prefix>/lib/infiniband; if you installed libmthca
somewhere else, set the OPENIB_DRIVER_PATH environment variable to
point to the directory with mthca.so.

To actually try things out, you can use the ibv_pingpong program
shipped as part of the libibverbs package.  For example, one one
system start the server side

    $ ibv_pingpong

and on another system start the client by passing the address of the
server (in this example I use IPv6 over IPoIB):

    $ ibv_pingpong fe80::202:c901:7fc:c711%ib0

The pingpong program has a number of options -- run ibv_pingpong -h to
see a list of the switches you can try.

The current code is stable for me, but all that means is that my tiny
selection of tests and test systems has not uncovered any of the bugs
that are undoubtedly present.  Some of the limitations I know about:

 - Only RC is implemented.  There are not even any functions to call
   to create UD address handles yet.
 - Only Tavor mode is supported -- PCI Express HCAs will not work if
   they are running mem-free firmware.
 - On x86, only CPUs with SSE will work now.  I'd be surprised if
   anyone has x86 system with an HCA that doesn't have SSE.

Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
kernels -- I don't expect any portability problems but I haven't even
built for other architectures.

In any case, please give this a spin and let me know how it looks to you.

My short- and medium-term plans are:

  1. Catch up on reviewing and applying the patche queue I'm sitting on.
  2. Land the Arbel mem-free mode support from the roland-uverbs branch
     onto the main trunk (and merge it upstream once 2.6.11 is out and
     2.6.12 opens).
  3. Implement UD support for userspace.  I should have this done before
     the end of next week.
  4. Implement mem-free support for userspace.

Thanks,
  Roland