[openib-general] IP addressing on InfiniBand networks

[James Lentini]

I'd like to summarize the discussion we've been having around 
addressing and start a new email thread with a more appropriate title.

First off, here is there requirement we are trying to satisfy:

   kDAPL consumers use an Internet Protocol (IP) addresses to
   identify remote nodes in an interoperable way.

   On the active side of a connection, an InfiniBand kDAPL provider must
   determine the IP address corresponding IB address (see the
   dat_ep_connect() function).

   On the passive side of a connection, a InfiniBand kDAPL provider must
   determine a source IB address for an InfiniBand connection request.
   This information can be obtain by a kDAPL consumer either in the
   DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or
   dat_cr_query()'s dat_cr_param structure.

   By interoperable, we mean that the solution must not introduce a
   non-standard protocol or force ULPs using kDAPL to perform special
   operations when using an InfiniBand network.

The first sentence of this requirement is the most important. This 
is to best support the existing practice of ULPs that use IP addresses: 
NFS, iSCSI, and and sockets applications. Administrators expect to be 
able to configure systems using IP addresses.

Note that this feature will be used. There has been considerable 
discussion on how NFS-RDMA software would make use of this feature. 
Other ULPs may use this feature as well.

Of the two mappings, the second (IB address to IP address) has proven 
the most difficult to implement. Here are the different 
implementations that have been proposed and a brief critique of each:

+ CM Private Data

    The active side of an IB connection could place its source IP
    address in the CM's private data. The passive side would retrieve
    the source IP from this location.

    Analysis: This approach introduces a new protocol that is not
    hidden from the ULP. The problem becomes where in the software
    stack this would be implemented. If it was implemented in the DAT
    provider, kDAPL consumers would not be able to communicate fully
    with non-kDAPL consumers (the non-kDAPL consumers would not
    be expecting the IP address and thus interpret the private data
    incorrectly). If this were implemented in a ULP, the ULP would
    not be able to use the same code for both an IB interface and an
    RNIC interface.

    The security of this is very week. An end node could easily present
    a false IP address.

+ Address Translation Service (ATS)

    Each IB node places a record containing its IP address and IB
    address in the SA database. kDAPL consults these records to
    map between addresses.

    Analysis: This requires all IB nodes to implement a new
    protocol/adhere to a new convention. The advantage is that ULPs do
    not need to be aware of it.

    Since a GID can have multiple IP address, there may be multiple
    records with the same GID. The passive side would need a convention
    for assuming which of these matches to the active side's IP.

    The security of this solution is also fairly weak. The end nodes
    must be trusted to place valid records in the SA. Since the
    configuration of IP addresses will occur on the end nodes, it would
    be difficult for the SA to validate the records. However, the SA is
    a central part of the network, so it may be possible to add additional
    security features if needed.

+ IPoIB

    IPoIB encapsulates IP packets in InfiniBand messages. There have been
    proposals to use the address resolution mechanisms in IPoIB to
    implement these features. IPv4 subnets use ARP and IPv6 subnets use
    Neighbor Discovery.

    Analysis:

    IPoIB is not free. All nodes would be need to implement it for
    this to work.

    The IB address -> IP address mapping on the passive side is
    problematic. If a reverse lookup were available, IPoIB would require
    both a GID and QP number as input. The passive side would know the GID
    but the QP number.

    Further more, reverse lookup is not well supported. On IPv4 subnets,
    RARP is quickly becoming (already?) obsolete. Neighbor Discovery
    doesn't support reverse lookup at all. [RFC 2461]

    In addition to all this, IPoIB restricts an IP subnet to the same scope
    as an IB subnet. If a kDAPL consumer desired to communicate between
    IB subnet's, IPoIB may not be sufficient.

+ GID as an IPv6 Address

    See the attachment to Caitlin Bestler's email:

    http://openib.org/pipermail/openib-general/2005-June/008104.html

    Analysis:

    This has been the least discussed option. One issue is
    that GIDs may not be easy to administer. GIDs can be specific
    to a particular channel adapter since they can contain EUI-64
    identifiers. Administrators avoid using Ethernet MAC addresses
    in configuration files and they should be able to avoid using
    adapter specific IB addresses as well. Another issue is how
    dynamically assigned SM GIDs would be managed.

Are there other implementations? Is there a way to more tightly 
integrate IP addressing with InfiniBand?