[ofa-general] Compiled IB packages

[Christopher Tanner]

Thanks Vladimir - very helpful. However, I'm running into a problem  
with compiling the ofa package. First, I had to specify the source  
location on the command line (Ubuntu puts it in a different place than  
RedHat or SUSE):

$ ./configure --kernel-sources=/usr/src/linux-source-2.6.24 ... (other  
stuff)

I'm getting this error:

   ERROR: Kernel configuration is invalid.
          include/linux/autoconf.h or include/config/auto.conf are  
missing.
          Run 'make oldconfig && make prepare' on kernel src to fix it.

This is confusing b/c both of those files exist.
$ locate autoconf.h
/usr/src/linux-headers-2.6.24-19-generic/include/linux/autoconf.h

$ locate auto.conf
/usr/src/linux-headers-2.6.24-19-generic/include/config/auto.conf

There's a whole bunch more errors that I assume spawn because of this  
initial error. The output from 'make' is attached (it's pretty long).  
Let me know what you think. Thanks!

-------------------------------------------
Chris Tanner
Space Systems Design Lab
Georgia Institute of Technology
christopher.tanner at gatech.edu
-------------------------------------------



On Sep 9, 2008, at 11:03 AM, Vladimir Sokolovsky wrote:

> Christopher Tanner wrote:
>> I am setting up a 16-node (homogeneous) cluster running Ubuntu 8.04  
>> server with Mellanox Infiniband cards. I downloaded (from the  
>> OpenFabrics website), compiled, and installed the following IB  
>> packages on the master node into the /usr/local/lib directory. The / 
>> usr/local directory is being shared to all of the nodes via NFS.  
>> All packages seemed to compile and install fine.
>> libibverbs
>> librdmacm
>> libibcm
>> libipathverbs
>> dapl
>> compat-dapl
>> libmlx4
>> libmthca
>> libcxgb3
>> libibcommon
>> libibumad
>> libibmad
>> opensm
>> infiniband-diags
>> I have a few questions:
>> a) Do I need to run 'make install' on each node or just the master  
>> node? All of the libraries in /usr/local/lib are visible to all  
>> nodes... Stated another way, does 'make install' put files  
>> elsewhere beside the /usr/local/lib directory? Does it alter OS  
>> configuration files to tell it to look for certain files in /usr/ 
>> local/lib?
>
> No, all the packages above will put their files under /usr/local
>
>> b)  I know I need to load the IB kernel modules (mlx4_core,  
>> mlx4_ib, rdma_ucm, ib_core, ib_mad, ib_mthca, ib_umad, ib_uverbs)  
>> in order for the IB cards to work. Are these compiled and installed  
>> with the above packages? Where does the kernel know where to look  
>> for modules? (Sorry, this question is very similar to the first one).
>
> The packages above are user space libraries/binaries. To install  
> kernel
> modules you should download the latest version of the ofa_1_4_kernel  
> tgz file from:
>
> http://www.openfabrics.org/downloads/ofa_1_4_kernel/
> To install, run:
> ./configure --with-core-mod --with-user_mad-mod --with-user_access- 
> mod --with-addr_trans-mod --with-mthca-mod --with-mthca_debug-mod -- 
> with-mlx4-mod --with-mlx4_en-mod --with-mlx4_debug-mod --with-cxgb3- 
> mod --with-ehca-mod --with-ipoib-mod --with-ipoib_debug-mod (... ,  
> see --help)
> make
> make install
>
>
>> c) The OFED software stack contains some stuff that isn't available  
>> for source download (e.g. ib-bonding, ibsim, libsdp). Are these  
>> necessary for the IB network to operate correctly? Since I'm  
>> running Ubuntu, obviously the src.rpm file won't work...
>
> All OFED tgz files that are available under:
> http://www.openfabrics.org/~vlad/ofed_1_4/SOURCES/
>
> ib-bonding source RPM can be downloaded from (you can open it to get  
> tgz file using cpio, if you need):
> http://www.openfabrics.org/~monis/ofed_1_4/
>
> This packages are not necessary for the IB network to operate  
> correctly, but
> it depends on what are you planning to do.
>
> Regards,
> Vladimir
>
>> Thanks to all for you help. Previous responses regarding issues  
>> with OpenSM worked great.
>> -------------------------------------------
>> Chris Tanner
>> Space Systems Design Lab
>> Georgia Institute of Technology
>> christopher.tanner at gatech.edu
>> -------------------------------------------