[openib-general] RDMA connection and address translation API

[Tom Tucker]

> -----Original Message-----
> From: Roland Dreier [mailto:rolandd at cisco.com] 
> Sent: Wednesday, August 24, 2005 4:03 PM
> To: Tom Tucker
> Cc: Sean Hefty; Roland Dreier; openib-general at openib.org
> Subject: Re: [openib-general] RDMA connection and address 
> translation API
> 
>     Tom> The issue is that this connection will be established when
>     Tom> the server may only want to accept requests that are
>     Tom> targetted to the 10.10.1.1 address.  I don't get why this is
>     Tom> such a big deal. You can preclude this behavior by simply
>     Tom> keeping a one to one mapping between the IPv4 addresses and
>     Tom> the GIDs using the existing protocols and without mandating a
>     Tom> private data format across *all* ulps and transports.
> 
> Well, a few problems with what you say:
> 
>  - ATS does not help at all with the case of a multi-homed interface.
>    Unless the remote system puts the IP it's trying to connect to
>    somewhere in the connection request, there is no way to be psychic
>    and recover this information.

I thought a single HCA could have multiple GIDs. All 
I'm advocating is that a "correct" multi-homed configuration 
has a one-to-one mapping between it's IP addresses and it's GIDS.

> 
>  - Mandating ATS use is dictating protocol design just as much as
>    requiring the CM private data to carry source and destination IP
>    addresses.

I think ATS dictates the kinds of authentication that can be 
done by the server over an IB transport, but not the protocol 
design. Certainly the private data can have additional 
authentication data (which I think is what you're advocating). 

> 
>  - It's not just preventing connections to the wrong local address.
>    NFS-RDMA wants the remote source address (ie getpeername()) so that
>    it can look it up in the exports list.

Agreed. But you could also get rid of ATS by allowing GIDs to 
be specified in the exports file and then treating them like 
IPv6 addresses for the purpose of subnet comparisons.

> 
>  - Saying that a given GID may only have a single IP address is
>    definitely a case of the cure being worse than the disease.  I
>    don't think we can forbid perfectly valid multi-homed
>    configurations just because it's inconvenient for us to 
> support them.

I think our different perspectives come from what 
we consider to be "perfectly valid multi-homed configurations". 
One approach advocates overloading private data, the other 
advocates overloading address assignments. 

My approach suffers from the fact that multiple IP addresses
for the same GID are just aliases that are interchangeable and at 
the remote end indistinguishable. The private data approach 
suffers from the need to mandate private data formats across 
all ulps and transports.

I prefer the former limitation/cost. 

> 
> By the way, as far as I can tell, there is NO formal 
> documentation of the NFS-RDMA wire protocol.  The current 
> draft (draft-ietf-nfsv4-rpcrdma-01.txt) simply says:
> 
>      This protocol is designed to function with equivalent semantics
>      over all appropriate RDMA transports.  In its abstract form, this
>      protocol does not implement RDMA directly. [...]  It therefore
>      becomes a useful, implementable standard when mapped onto a
>      specific RDMA transport, such as iWARP [RDDP] or Infiniband [IB].
> 
>      [...]
> 
>      In setting up a new RDMA connection, the first action by an RPC
>      client will be to obtain a transport address for the server.  The
>      mechanism used to obtain this address, and to open an RDMA
>      connection is dependent on the type of RDMA transport, 
> and outside
>      the scope of this protocol.
> 
> So it seems perfectly reasonable and acceptable for the 
> mapping of NFS-RDMA onto IB to specify that the source and 
> destination IP addresses for an IB connection are placed in 
> the CM private data.
> This seems much easier than trying to turn ATS into an IETF standard.
> 
>  - R.
> 

I think there is a way to get rid of ATS as I described above without 
overloading the private data.

Phew -- I'm exhausted. I'm going to go write code ;-)