[ofa-general] Compiled IB packages

[Christopher Tanner]

I am setting up a 16-node (homogeneous) cluster running Ubuntu 8.04  
server with Mellanox Infiniband cards. I downloaded (from the  
OpenFabrics website), compiled, and installed the following IB  
packages on the master node into the /usr/local/lib directory. The / 
usr/local directory is being shared to all of the nodes via NFS. All  
packages seemed to compile and install fine.

libibverbs
librdmacm
libibcm
libipathverbs
dapl
compat-dapl
libmlx4
libmthca
libcxgb3
libibcommon
libibumad
libibmad
opensm
infiniband-diags

I have a few questions:
a) Do I need to run 'make install' on each node or just the master  
node? All of the libraries in /usr/local/lib are visible to all  
nodes... Stated another way, does 'make install' put files elsewhere  
beside the /usr/local/lib directory? Does it alter OS configuration  
files to tell it to look for certain files in /usr/local/lib?

b)  I know I need to load the IB kernel modules (mlx4_core, mlx4_ib,  
rdma_ucm, ib_core, ib_mad, ib_mthca, ib_umad, ib_uverbs) in order for  
the IB cards to work. Are these compiled and installed with the above  
packages? Where does the kernel know where to look for modules?  
(Sorry, this question is very similar to the first one).

c) The OFED software stack contains some stuff that isn't available  
for source download (e.g. ib-bonding, ibsim, libsdp). Are these  
necessary for the IB network to operate correctly? Since I'm running  
Ubuntu, obviously the src.rpm file won't work...

Thanks to all for you help. Previous responses regarding issues with  
OpenSM worked great.

-------------------------------------------
Chris Tanner
Space Systems Design Lab
Georgia Institute of Technology
christopher.tanner at gatech.edu
-------------------------------------------