[ofa-general] Compiled IB packages

[Vladimir Sokolovsky]

Christopher Tanner wrote:
> I am setting up a 16-node (homogeneous) cluster running Ubuntu 8.04 
> server with Mellanox Infiniband cards. I downloaded (from the 
> OpenFabrics website), compiled, and installed the following IB packages 
> on the master node into the /usr/local/lib directory. The /usr/local 
> directory is being shared to all of the nodes via NFS. All packages 
> seemed to compile and install fine.
> 
> libibverbs
> librdmacm
> libibcm
> libipathverbs
> dapl
> compat-dapl
> libmlx4
> libmthca
> libcxgb3
> libibcommon
> libibumad
> libibmad
> opensm
> infiniband-diags
> 
> I have a few questions:
> a) Do I need to run 'make install' on each node or just the master node? 
> All of the libraries in /usr/local/lib are visible to all nodes... 
> Stated another way, does 'make install' put files elsewhere beside the 
> /usr/local/lib directory? Does it alter OS configuration files to tell 
> it to look for certain files in /usr/local/lib?
> 

No, all the packages above will put their files under /usr/local

> b)  I know I need to load the IB kernel modules (mlx4_core, mlx4_ib, 
> rdma_ucm, ib_core, ib_mad, ib_mthca, ib_umad, ib_uverbs) in order for 
> the IB cards to work. Are these compiled and installed with the above 
> packages? Where does the kernel know where to look for modules? (Sorry, 
> this question is very similar to the first one).
> 

The packages above are user space libraries/binaries. To install kernel
modules you should download the latest version of the ofa_1_4_kernel tgz file from:

http://www.openfabrics.org/downloads/ofa_1_4_kernel/
To install, run:
./configure --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mthca_debug-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_debug-mod 
--with-cxgb3-mod --with-ehca-mod --with-ipoib-mod --with-ipoib_debug-mod (... , see --help)
make
make install


> c) The OFED software stack contains some stuff that isn't available for 
> source download (e.g. ib-bonding, ibsim, libsdp). Are these necessary 
> for the IB network to operate correctly? Since I'm running Ubuntu, 
> obviously the src.rpm file won't work...
> 

All OFED tgz files that are available under:
http://www.openfabrics.org/~vlad/ofed_1_4/SOURCES/

ib-bonding source RPM can be downloaded from (you can open it to get tgz file using cpio, if you need):
http://www.openfabrics.org/~monis/ofed_1_4/

This packages are not necessary for the IB network to operate correctly, but
it depends on what are you planning to do.

Regards,
Vladimir

> Thanks to all for you help. Previous responses regarding issues with 
> OpenSM worked great.
> 
> -------------------------------------------
> Chris Tanner
> Space Systems Design Lab
> Georgia Institute of Technology
> christopher.tanner at gatech.edu
> -------------------------------------------