[openib-general] RE: [dat-discussions] socket based connectionmodel for IB proposal - round 3

[Kanevsky, Arkady]

The goal that this proposal is to provide underpinning for common RDMA
transport CM.
Thus, the API ULP (both user space and kernel space) use socket
addressing.

For ULP addressing this means 5 tuple: protocol, src IP addr, src port,
dst IP addr,
and dst port. Port is 16 bit entity.

The proposal just provide a mechanism for exchanging this 5-tuple
between two sides.

Which entity is responsible to "use" the proposed protocol is an
interesting one.
I was assuming that this will be CM. After all the proposed protocol is
CM extension
protocol.
But it can be another entity module between CM and ULP.
Its job will be taking 5 tuple and populating private data and
converting dst port to SID. Since OpenIB addr.c already deals with IP to
IB address translations it is a logical candidate for it. On remote side
it extracts info from private data and populates socket info for
Consumer and passes Consumer a pointer to Consumer private data.

Another interesting place to deal with is listening point.
Since it is common RDMA API, 16 bit port should be use for it also.
This means that the same module should  locally convert port to IB SID
before passing it
to CM.
CM just ensures that incoming connection request which matches listening
SID.

While it is possible to do wildcarding on the whole SID, I had not seen
it is used selectively on individual bits of a SID or a port.

While SDP does the conversion to IB SID from Ethernet port, this
proposal
shift the responsibility for port and IP address conversion from ULP
down.

Now lets look at each field proposed to be moved from protocol private
data to SID.

Protocol version. This mean that in the future if protocol version will
be
bumped up we will have to change the SID on which Consumer listens on
and
requests sent to. Not sure how to do that without changing ULP. Does not
look
like a good idea.

IP version. This can be incorporated into SID. But if HCA has multiple
IP addresses
assigned to it the listening point need to specify its IP address(es).
The current verbs and/or API will have to be changed to support it.
But if socket is passed to listen on it does have all the needed info.
Looks fine.

Ethernet Protocol. The same as the one above.

Src port. Very questionable. For that listening SID must have wild card
for portion
of SID where SRC port is incorporated. Since ULP is not aware or ever
see it,
it is possible. But this pushes the definition of SID beyond it current
IBTA spec
statement of "similar to TCP port number". The query of listen point
should also
hide the wildcarded SID in this case.

DAPL APIs (uDAPL and kDAPL) does not expose local IP address for listen
point.
An additional API can be added to support passing local socket to listen
on
instead of Connection Qualifier. Since it is addition no backwards
compatibility issues.
The current ULPs/Apps will still use "the default API address" and the
protocol assigned SID as connection qualifier.

The new API ensures that locally SID conversion takes place.
The use of protocol defined range of SIDs ensures that remote side knows
to parse
private data according to proposed protocol format.

Arkady


Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
275 Totten Pond Rd.                  Fax: 781-895-1195
Waltham, MA 02451-2010          central phone: 781-768-5300
 

> -----Original Message-----
> From: Sean Hefty [mailto:mshefty at ichips.intel.com] 
> Sent: Friday, November 11, 2005 12:43 PM
> To: Kanevsky, Arkady
> Cc: Sean Hefty; swg at infinibandta.org; 
> dat-discussions at yahoogroups.com; openib-general at openib.org
> Subject: Re: [openib-general] RE: [dat-discussions] socket 
> based connectionmodel for IB proposal - round 3
> 
> Kanevsky, Arkady wrote:
> > So what you are proposing is that Listener will specify 
> IETF port (2 
> > bytes).
> > CM will generate an IB SID to listen on. That SID will have 
> > wildcarding for 24 bits.
> > The requestor will specify: version, IP version, SRC port 
> and DST port.
> > Based on that CM will generate the SID to send request to.
> 
> No, the listener or requester generate the SID, not the IB CM 
> - the same way SDP works today.
> 
> > It will also encode IP addresses into Private data based on 
> IP version.
> > 
> > This makes IP addresses, SIDs and private data format 
> interdependent 
> > and not orthogonal which it is now.
> > It also changes the meaning of SID which currently has a meaning of 
> > TCP port.
> 
> I'm not proposing this.  I'm merely stating that is is a 
> valid option to consider.  The private data format and SIDs 
> are not orthogonal anyway.  The port number's embedded in the 
> SID, and the SID indicates the format of the private data.  
> They are interdependent by definition.
> 
> If it's okay to put the destination port number in the SID, 
> why not the protocol type, or IP version?
> 
> > It also does not allow to use the private data formating 
> for other SIDs.
> 
> Private data is private.  It should not be owned, set, 
> interpreted, modified, or touched by the CM.  It's up to the 
> service to define and use.
> 
> What's this proposal defines is basically a 65th bit for the 
> service ID.  If the new 65 bit SID is:
> 
> 1 <anything>	- private data has this format
> 0 <anything>	- private data format is unknown
> 
> Why do we need this 65th bit?
> 
> > It looks like a big hack. Is it worth it for extra 4 bytes 
> of private 
> > data for Consumers?
> 
> It's a trade off between SID space and private data.  
> Consumers need to decide how important those extra 4 bytes are.
> 
> - Sean
> 
>