[openib-general] IB Address Translation service

[Libor Michalek]

On Mon, Feb 28, 2005 at 10:29:19AM -0800, Roland Dreier wrote:
>     Roland> First, let's understand the problem we're trying to solve.
>     Roland> Who are the consumers of this address translation service?
> 
>     shaharf> Any ULPs at user & kernel, and also some
>     shaharf> applications.
> 
> I think this is too general an answer.  We should be designing based
> on specific ULPs and applications.  For example, I don't see anything
> particularly useful to IPoIB in this API.  Perhaps Libor can comment
> on how this API works for SDP.

  SDP does implement a subset of the proposed functionality for
resolving IP addresses to PathRecords which can then be used in
a CM REQ request, plus some basic caching. All the code is isolated
to a single file, sdp_link.c. There's really only a single entry
point API, plus a completion function:

int sdp_link_path_lookup(u32 dst_addr,
			 u32 src_addr,
			 int bound_dev_if,
			 void (*completion)(u64 id,
					    int status,
					    u32 dst_addr,
					    u32 src_addr,
					    u8  hw_port,
					    struct ib_device *ca,
					    struct ib_sa_path_rec *path,
					    void *arg),
			 void *arg,
			 u64  *id);

  The values are based on strictly what is needed by either the Linux
routing code to resolve the address, or the IB APIs to establish the
connection. The implementation has three stages:

  - src/dst IP address -> IPoIB net_device, IB ca, IB port, IB pkey.
  - dst IP address and IPoIB net_device -> dst GID using IPoIB ARP
  - dst GID -> PathRecord using ib_sa.

  A cancel function based on the 'id' parameter would be a nice to have
but is not strictly necessary, since the lookup will eventually compelte
one way or another and any dead connection will be cleaned up at that
point. 

>     shaharf> My take right now is to implement a kernel based
>     shaharf> mechanism and a user mode library to interface it.  There
>     shaharf> are other feasible solutions. I would really like you
>     shaharf> have your suggestions and preferences.
> 
> Unless there is a real kernel consumer that needs something this
> elaborate, I would prefer to implement this sort of caching service as
> a userspace daemon/library.  This allows for more sophisticated
> implementations (eg persistent caches) and also makes debugging and
> maintenance easier.
> 
>     shaharf> I think that starting with the APIs is a valid approach
>     shaharf> that has its own advantages and disadvantages.
>  
> Sure, it's always good to have code in hand to start a discussion.
> But in this case the API seems to be far ahead of its consumers, so it
> ends up feeling overengineered to me.

  Maybe a list of near term ULPs and APIs and the functions that they
are very likely to need? All though it usually makes sens to add the
functionality to a bare framework as the ULPs and APIs are added to
avoid functionality for a ULP/API that never ends up making it to the
kernel...

  Also, I'm not too excited about mixing IP addressing with ATS.


-Libor