[openib-general] IB Address Translation service

[Libor Michalek]

On Mon, Feb 28, 2005 at 11:55:54PM +0200, Yaron Haviv wrote:
> From:  Libor Michalek
> > 
> >   SDP does implement a subset of the proposed functionality for
> > resolving IP addresses to PathRecords which can then be used in
> > a CM REQ request, plus some basic caching. All the code is isolated
> > to a single file, sdp_link.c. There's really only a single entry
> > point API, plus a completion function:
> > 
> > int sdp_link_path_lookup(u32 dst_addr,
> > 			 u32 src_addr,
> > 			 int bound_dev_if,
> > 			 void (*completion)(u64 id,
> > 					    int status,
> > 					    u32 dst_addr,
> > 					    u32 src_addr,
> > 					    u8  hw_port,
> > 					    struct ib_device *ca,
> > 					    struct ib_sa_path_rec *path,
> > 					    void *arg),
> > 			 void *arg,
> > 			 u64  *id);
> > 
> >   The values are based on strictly what is needed by either the Linux
> > routing code to resolve the address, or the IB APIs to establish the
> > connection. The implementation has three stages:
> > 
> >   - src/dst IP address -> IPoIB net_device, IB ca, IB port, IB pkey.
> >   - dst IP address and IPoIB net_device -> dst GID using IPoIB ARP
> >   - dst GID -> PathRecord using ib_sa.
> 
> Libor the idea is that ib_at provides similar functionality 
> Sahar looked through your SDP code prior to proposing the API
> We would like to have a common API for all the ULP's that provide that
> functionality, and specifically now when we implement kDAPL over OpenIB.

  Sure, it does make sense to break this code into it's own module if
there are multiple ULPs that need to use the code, and sounds like we
are getting close to having another ULP which needs this resolution.
However, the API feels like it is intended to provide every possible
bell and whistle imaginable. It is far better to start with a simple
clean minimum of features and add to the functionality as new ULPs are 
introduced or old ULPs are improved. I may be wrong, and people have
intentions for each function and parameter that you proposed, but it
feels so large that it would be good to hear which ULPs you envision
using each of the functions, especially some of the less obvious ones.
Remember, this API does not need to be frozen out of the gate, changes
can and will be made, incompatabilities will be introduced.

  I would like to see the feature set, if possible, split between
user and kernel space, we should minimize what's in the kernel, and
features that are only needed in userspace, should only be implemented
in userspace.

  I also see kDAPL as weak justification for a feature. (notice I did
no say uDAPL) I would be better to see a kDAPL proposal, by which I
mean code, that had a chance before we start including features for
it in surrounding code. As it stands it has an uphill battle, and not
just because of the API itself.

> To summaries the differences:
> 
> The reasons we broken it to two functions (IP->GID, GID->Path) and not
> have an IP->Path API (like we also used to have in our gen1 stack) are:
> 
> a. some consumers will only need the 1st part (e.g. just to know which
>    HCA to use)
> b. some may use only the 2nd part (e.g. IPoIB, SRP)
> c. you can get parameters from the first part (e.g. P_Key, and decide to
>    overwrite it with your own P_Key, etc')
> d. the 2nd function provides more options for multipath, partitioning,
>    QoS
> e. we can now more easily use different IP resolution mechanisms without
>    changing the 2nd function (ARP or ATS).  

  I have no real problem with spliting the two halves of the resolution
into two functions, as long as the common case of IP->Path is easy to
perform. By which I mean that all the parameters I need for GID->Path
are either in the IP->GID result or are obvious. Which it sounds like
from your later comment.

> We added source IP and TOS as optional parameters for the IP->GID, just
> because IP route can be defined for Src/dst/TOS, and it's already part
> of Linux.

  OK, sounds good. I'm using source IP now, since it's possible to bind
a socket to a specific source address before connecting.

-Libor