[openib-general] IB Address Translation service

[Yaron Haviv]

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Libor Michalek
> Sent: Monday, February 28, 2005 8:55 PM
> To: Roland Dreier
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> 
>   SDP does implement a subset of the proposed functionality for
> resolving IP addresses to PathRecords which can then be used in
> a CM REQ request, plus some basic caching. All the code is isolated
> to a single file, sdp_link.c. There's really only a single entry
> point API, plus a completion function:
> 
> int sdp_link_path_lookup(u32 dst_addr,
> 			 u32 src_addr,
> 			 int bound_dev_if,
> 			 void (*completion)(u64 id,
> 					    int status,
> 					    u32 dst_addr,
> 					    u32 src_addr,
> 					    u8  hw_port,
> 					    struct ib_device *ca,
> 					    struct ib_sa_path_rec *path,
> 					    void *arg),
> 			 void *arg,
> 			 u64  *id);
> 
>   The values are based on strictly what is needed by either the Linux
> routing code to resolve the address, or the IB APIs to establish the
> connection. The implementation has three stages:
> 
>   - src/dst IP address -> IPoIB net_device, IB ca, IB port, IB pkey.
>   - dst IP address and IPoIB net_device -> dst GID using IPoIB ARP
>   - dst GID -> PathRecord using ib_sa.
> 
>   A cancel function based on the 'id' parameter would be a nice to
have
> but is not strictly necessary, since the lookup will eventually
compelte
> one way or another and any dead connection will be cleaned up at that
> point.
> 

Libor the idea is that ib_at provides similar functionality 
Sahar looked through your SDP code prior to proposing the API
We would like to have a common API for all the ULP's that provide that
functionality, and specifically now when we implement kDAPL over OpenIB.

To summaries the differences:

The reasons we broken it to two functions (IP->GID, GID->Path) and not
have an IP->Path API (like we also used to have in our gen1 stack) are:

a. some consumers will only need the 1st part (e.g. just to know which
HCA to use)
b. some may use only the 2nd part (e.g. IPoIB, SRP)
c. you can get parameters from the first part (e.g. P_Key, and decide to
overwrite it with your own P_Key, etc')
d. the 2nd function provides more options for multipath, partitioning,
QoS
e. we can now more easily use different IP resolution mechanisms without
changing the 2nd function (ARP or ATS).  

We added source IP and TOS as optional parameters for the IP->GID, just
because IP route can be defined for Src/dst/TOS, and it's already part
of Linux.
we added multipath, IB QoS etc' because we have more than one
applications that need it today, e.g. people that want to run IPC/MPI
and NFS on the same fabric may want 2 separate VLs and SLs, some
applications need APM support, some applications need source based
routing, ..
Since there are commercial SMs that can provide all the advanced
capabilities, we want to enable the OpenIB stack to make use of it.

By default you can nest the 2 functions (call the first, and than use
its result to call the second), what you will get is that the ULP will
use the HCA/Port associated with the IPoIB subnet, will use the same
Partition as the IPoIB interface, and will use the same QoS/SL as IPoIB.
Optionally the consumer can put his own QoS, Partitioning, etc' for the
2nd function if he knows where to take it from.

An example of what you can get with the default mode:
You define few partitions in the fabric (central configuration), each
with its IP subnet and SL (from MCRecord).
And then just use different IP subnet for isolation (Partitions or VLs)
NO need for any manual/local configuration on the host side and your
IPC, Storage are now running on separate VLs and/or Partitions, what we
got is something we can explain to users and developers that haven't
read the IB 1000s page book, and don't sit regularly in IBTA meetings.

Yaron