[ofa-general] Infiniband and Ubuntu

[Jason Gunthorpe]

On Thu, Jul 23, 2009 at 05:27:23PM -0400, Mark J. Pearrow wrote:

> The short question is "is there a howto for getting Infiniband and Ubuntu 
> working?"

I've done this and other similar things, it isn't too hard..

I like to use 2.6.30 these days for ConnectX, this is the base kernel
for OFED 1.5 and has a few additional fixes over .28 that comes with
Jaunty, but you can do that after getting the user space setup. I
don't think there is anything fatal in .28..

When using stock kernels it is apparently best not to use OFED and its
patches.. You are going to want to find the following tar files:

infiniband-diags-1.5.1.tar.gz  libibmad-1.3.1.tar.gz    libmlx4-1.0.1.tar.gz
libibcm-1.0.4.tar.gz           libibumad-1.3.1.tar.gz   librdmacm-1.0.8.tar.gz
libibverbs-1.1.2.tar.gz        opensm-3.3.1.tar.gz

I usually just grab them from git rather that fussing:
 git://git.openfabrics.org/~shefty/libibcm.git
 git://git.kernel.org/pub/scm/libs/infiniband/libibverbs.git
 git://git.kernel.org/pub/scm/libs/infiniband/libmlx4.git
 git://git.openfabrics.org/~shefty/librdmacm.git
 git://git.openfabrics.org/~sashak/management.git [umad, mad, opensm and diags]

I like to install the entire set in an opt directory. Do the following
repeatedly:

cd XXXX
./autogen.sh 
LDFLAGS=-L/opt/ofa-1.5/lib CPPFLAGS=-I/opt/ofa-1.5/include ./configure --prefix=/opt/ofa-1.5
make
make install

In this order:
 libumad
 libibmad
 opensm
 libibverbs  (*)
 libmlx4     (*)
 libibcm
 librdmacm
 infiniband-diags

The ones with a * are already available through ubuntu, but mlx4
should be updated to 1.0.1. You can use 'debian/rules binary' instead
of the configure/make/make install sequence to produce a new binary
.deb for these two.

Once complete you'll have a /opt/ofa-1.5/ full of all the tools and
libraries you need.

Since the stock Jaunty kernel already includes the kernel side and pci
driver autoloading does most of the work all you need is to modprobe
a few additional things: ib_ucm ib_ipoib rdma_ucm

Thats basically it for the core IB.

Ah, one trick, depending on various factors you may not get rpath
support out of libtool. I forget how to fix this up - You really want
to build all of the above using rpath.. But barring that, you can do
'export LD_LIBRARY_PATH=/opt/ofa-1.5/lib' at the very start to make it
all work out.

The biggest problem was the ubuntu kernel version. You want at least
2.6.30 for ConnectX cards. Jaunty is only .28..

Oh, do the following to setup the ofa-1.5 directory to make it nice
and tidy:

mkdir -p /opt/ofa-1.5/share/man
(cd /opt/ofa-1.5/ && ln -s share/man man)

[Why the variance amongst the packages I have not tracked down]

There is probably also a way to do the above, but use the included
.spec files to make rpms and alien those to .debs but that has always
been too much trouble for me.

I'm sure lots of people would be grateful to have Debian packaging :)

Jason