[openib-general] IP addressing on InfiniBand networks

[James Lentini]

On Tue, 28 Jun 2005, Hal Rosenstock wrote:

> On Tue, 2005-06-28 at 15:24, James Lentini wrote:
>> + IPoIB
>>
>>     IPoIB encapsulates IP packets in InfiniBand messages. There have been
>>     proposals to use the address resolution mechanisms in IPoIB to
>>     implement these features. IPv4 subnets use ARP and IPv6 subnets use
>>     Neighbor Discovery.
>>
>>     Analysis:
>>
>>     IPoIB is not free. All nodes would be need to implement it for
>>     this to work.
>>
>>     The IB address -> IP address mapping on the passive side is
>>     problematic. If a reverse lookup were available, IPoIB would require
>>     both a GID and QP number as input. The passive side would know the GID
>>     but the QP number.
>>
>>     Further more, reverse lookup is not well supported. On IPv4 subnets,
>>     RARP is quickly becoming (already?) obsolete.
>
> The IPoIB HW address includes the QPN (in addition to the GID). This is
> also problematic.
>
>> Neighbor Discovery
>>     doesn't support reverse lookup at all. [RFC 2461]
>>
>>     In addition to all this, IPoIB restricts an IP subnet to the same scope
>>     as an IB subnet.
>
> IPoIB does not limit an IP subnet to an IB subnet. It can span IB
> subnets. However, IB routers were not completed in the IB architecture.

I found this limitation in section 3.3 of the "IP over 
InfiniBand(IPoIB) Architecture" draft (April, 2004 version,
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-04.txt)

         Until the above conditions are met it is not possible to
         implement IPoIB subnets that span IB subnets. The IPoIB
         standards have however been defined with this possibility in
         mind.

Am I looking at an old version of the spec?

>
>> If a kDAPL consumer desired to communicate between
>>     IB subnet's, IPoIB may not be sufficient.
>
> Are you referring to 2 disjoint IB subnets ?

Yes. Your point is valid. If there are no routers, there is no reason 
to worry about it.

> What about IB <-> iWARP ?

I hadn't been considering that kind of communication. My assumption is 
that if a translator was created for IB <-> iWARP it would solve this 
issue...actually, a hypothetical translator is likely to use the 
solution we choose.

>> + GID as an IPv6 Address
>>
>>     See the attachment to Caitlin Bestler's email:
>>
>>     http://openib.org/pipermail/openib-general/2005-June/008104.html
>>
>>     Analysis:
>>
>>     This has been the least discussed option. One issue is
>>     that GIDs may not be easy to administer. GIDs can be specific
>>     to a particular channel adapter since they can contain EUI-64
>>     identifiers. Administrators avoid using Ethernet MAC addresses
>>     in configuration files and they should be able to avoid using
>>     adapter specific IB addresses as well.
>
> If they don't like ethernet MACs, they really won't like GUIDs/GIDs
> as they are even longer.

Length aside, GUIDs/GIDs are not manageable like IP addresses.

>
>> Another issue is how
>>     dynamically assigned SM GIDs would be managed.
>
> Do you mean SM (assigned additional) GUIDs ?

No, I was referring to the SM assigned GIDs described in property 3c 
of section 4.1.1 of the IB spec.

>
> -- Hal
>