[openib-general] IP addressing on InfiniBand networks

[David M. Brean]

Hello,

Comments inline.

James Lentini wrote:

>
> I'd like to summarize the discussion we've been having around 
> addressing and start a new email thread with a more appropriate title.
>
> First off, here is there requirement we are trying to satisfy:
>
>   kDAPL consumers use an Internet Protocol (IP) addresses to
>   identify remote nodes in an interoperable way.
>
>   On the active side of a connection, an InfiniBand kDAPL provider must
>   determine the IP address corresponding IB address (see the
>   dat_ep_connect() function).
>
>   On the passive side of a connection, a InfiniBand kDAPL provider must
>   determine a source IB address for an InfiniBand connection request.
>   This information can be obtain by a kDAPL consumer either in the
>   DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or
>   dat_cr_query()'s dat_cr_param structure.
>
>   By interoperable, we mean that the solution must not introduce a
>   non-standard protocol or force ULPs using kDAPL to perform special
>   operations when using an InfiniBand network.
>
> The first sentence of this requirement is the most important. This is 
> to best support the existing practice of ULPs that use IP addresses: 
> NFS, iSCSI, and and sockets applications. Administrators expect to be 
> able to configure systems using IP addresses.
>
> Note that this feature will be used. There has been considerable 
> discussion on how NFS-RDMA software would make use of this feature. 
> Other ULPs may use this feature as well.
>
> Of the two mappings, the second (IB address to IP address) has proven 
> the most difficult to implement. Here are the different 
> implementations that have been proposed and a brief critique of each:
>
> + CM Private Data
>
>    The active side of an IB connection could place its source IP
>    address in the CM's private data. The passive side would retrieve
>    the source IP from this location.
>
>    Analysis: This approach introduces a new protocol that is not
>    hidden from the ULP. The problem becomes where in the software
>    stack this would be implemented. If it was implemented in the DAT
>    provider, kDAPL consumers would not be able to communicate fully
>    with non-kDAPL consumers (the non-kDAPL consumers would not
>    be expecting the IP address and thus interpret the private data
>    incorrectly). If this were implemented in a ULP, the ULP would
>    not be able to use the same code for both an IB interface and an
>    RNIC interface.
>

I think the DAT provider should fill in the CM
private-data area with the source IP address (and a bit of additional)
information.  So, ULPs aren't participating in this aspect of IB
communication.  Also, it should be possible to have a DAT ULP
communicate with a peer that isn't implemented using DAT as long as the
location and format of the source IP address (and additional)
information in the CM private-data area is specified.  For example,
there is a draft "iSER on IB" document that specifies the format of
additional information sent in the CM private-data area.  Similar
document could be written for NFS using RDMA and other ULPs. [Perhaps
there should be an IBA defined structure for this purpose.]  For ULPs
written for DAT, this DAT provider does this for the ULP.


>    The security of this is very week. An end node could easily present
>    a false IP address.
>

If the DAT provider is responsible for providing this information where
DAT is used, the security is no worse than IP.


> + Address Translation Service (ATS)
>

The ATS proposal doesn't describe how to select the P_Key that must be
specified when installing the SA record.  In cases where IPoIB is
running on the IB subnet and the IP address is bound to an IPoIB link
interface, it's possible to query the IP stack to determine the P_Key
associated with the IP address.  In cases where IPoIB isn't used on the
IB subnet, it's not clear where the IP address and P_Key come from.
[I've been told that one solution is to use the IP address bound to an
ethernet port and install an SA entry for every IB partition that the
node can access based on the contents of its local P_Key table in the HCA.]

If an IB fabric chooses not to run IPoIB, perhaps this is a scenario
where the IB GID should be used as the IPv6 address as others have
suggested.  As a result, ATS is not needed.

ATS also requires a mechanism on each IB node that monitors the status
of IP addresses and updates the SA state.

>    Each IB node places a record containing its IP address and IB
>    address in the SA database. kDAPL consults these records to
>    map between addresses.
>
>    Analysis: This requires all IB nodes to implement a new
>    protocol/adhere to a new convention. The advantage is that ULPs do
>    not need to be aware of it.
>
>    Since a GID can have multiple IP address, there may be multiple
>    records with the same GID. The passive side would need a convention
>    for assuming which of these matches to the active side's IP.
>
>    The security of this solution is also fairly weak. The end nodes
>    must be trusted to place valid records in the SA. Since the
>    configuration of IP addresses will occur on the end nodes, it would
>    be difficult for the SA to validate the records. However, the SA is
>    a central part of the network, so it may be possible to add additional
>    security features if needed.
>

It is possible to have the SA authenticate any IB client attempting to
install an entry in the SA.  The SA record is protected by a Service
Key.  By default, the Service Key value is probably zero - no
protection.  However, the IB administrator can assign a value to the
Service Key associated with a Service Name.  ATS specifies a well-known
Service Name of "DAPL Address Translation Service", so the ATS
specification could describe steps needed to assign a Service Key value
for it.  Also, this Service Key would need to be securely distributed to
all the IB clients that have authority to enter SA records with the
Service Name for ATS.

-David

> + IPoIB
>
>    IPoIB encapsulates IP packets in InfiniBand messages. There have been
>    proposals to use the address resolution mechanisms in IPoIB to
>    implement these features. IPv4 subnets use ARP and IPv6 subnets use
>    Neighbor Discovery.
>
>    Analysis:
>
>    IPoIB is not free. All nodes would be need to implement it for
>    this to work.
>
>    The IB address -> IP address mapping on the passive side is
>    problematic. If a reverse lookup were available, IPoIB would require
>    both a GID and QP number as input. The passive side would know the GID
>    but the QP number.
>
>    Further more, reverse lookup is not well supported. On IPv4 subnets,
>    RARP is quickly becoming (already?) obsolete. Neighbor Discovery
>    doesn't support reverse lookup at all. [RFC 2461]
>
>    In addition to all this, IPoIB restricts an IP subnet to the same 
> scope
>    as an IB subnet. If a kDAPL consumer desired to communicate between
>    IB subnet's, IPoIB may not be sufficient.
>
> + GID as an IPv6 Address
>
>    See the attachment to Caitlin Bestler's email:
>
>    http://openib.org/pipermail/openib-general/2005-June/008104.html
>
>    Analysis:
>
>    This has been the least discussed option. One issue is
>    that GIDs may not be easy to administer. GIDs can be specific
>    to a particular channel adapter since they can contain EUI-64
>    identifiers. Administrators avoid using Ethernet MAC addresses
>    in configuration files and they should be able to avoid using
>    adapter specific IB addresses as well. Another issue is how
>    dynamically assigned SM GIDs would be managed.
>
> Are there other implementations? Is there a way to more tightly 
> integrate IP addressing with InfiniBand?