[openib-general] DHCP over infiniband

[Hal Rosenstock]

Hi Cynthia,

On Fri, 2005-09-23 at 13:25, cynthia segura wrote:
> I'm new to infiniband and I'm trying to assign an IP address to an 
> infiniband interface using DHCP. Currently, I have a small test cluster 
> (two nodes only) each with an infiniband HCA cabled back to back (no 
> switch).  I am running a  2.6.12.4 kernel with infiniband support 
> compiled in the kernel. When the nodes boot, I can see both of the 
> infiniband interfaces and can assign them IP addresses using ifconfig. 

The DHCP client in the kernel does not support IPoIB. You need to
ifconfig the interfaces as you are doing. There were modifications to
the ISC DHCP client and server for IPoIB posted a long time ago on the
list but that would need dusting off again to make it work again. I'm
not sure if that is of interest.

> However, it appears that there is not communication between the 
> interfaces (I can't ping them).

It sounds like you are run in "back to back" HCA mode.

>  I consulted the FAQ 
> (http://www.openib.org/docs/ipoib_faq.txt) and it appear that I need a 
> subnet manager running on at least one of the nodes,

Yes.
 
>  so I downloaded and 
> installed opensm. However, when I try to run it, I receive the following 
> error:
-------------------------------------------------
> OpenSM Rev:openib-1.1.0
> Command Line Arguments:
>  Log File: /var/log/osm.log
> -------------------------------------------------
> OpenSM Rev:openib-1.1.0
> 
> warn: [21042] umad_init: wrong ABI version: 
> /sys/class/infiniband_mad/abi_version is 2 but library ABI is 5

Do you have an old user_mad.c ? I forget what was in 2.6.12.4 but maybe
there were some ABI changes. I can confirm if needed.

> Using default guid 0x5ad000004b4f1
> SM port is down.

Is the SM port cabled to anything ?

> So, then I assumed that I need to recompile the infiniband modules with 
> the latest OpenIB stack, so I download it and followed these instructions:
> https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet
> 
> Unfortunately, I when I try to compile the 2.6.12.4 kernel with 
> infiniband support:
> 
> #
> # InfiniBand support
> #
> CONFIG_INFINIBAND=y
> CONFIG_INFINIBAND_USER_MAD=y
> CONFIG_INFINIBAND_USER_ACCESS=y
> CONFIG_INFINIBAND_MTHCA=y
> # CONFIG_INFINIBAND_MTHCA_DEBUG is not set
> CONFIG_INFINIBAND_IPOIB=y
> # CONFIG_INFINIBAND_IPOIB_DEBUG is not set
> # CONFIG_INFINIBAND_SDP is not set
> # CONFIG_INFINIBAND_SRP is not set
> 
> I receive the following errors:
> 
> drivers/built-in.o(.init.text+0x533c): In function `ib_uat_init':
> : undefined reference to `class_create'
> drivers/built-in.o(.init.text+0x5385): In function `ib_uat_init':
> : undefined reference to `class_device_create'
> drivers/built-in.o(.exit.text+0x802): In function `ib_uat_cleanup':
> : undefined reference to `class_device_destroy'
> drivers/built-in.o(.exit.text+0x810): In function `ib_uat_cleanup':
> : undefined reference to `class_destroy'
> make: *** [.tmp_vmlinux1] Error 1

There is a backpatch patch available for this:
https://openib.org/svn/gen2/branches/backport/2.6.12/uat_3465_to_2_6_12.patch

Hope this helps to get you up and running.

-- Hal