[ofa-general] Re: iWARP peer-to-peer CM proposal

[Caitlin Bestler]

> -----Original Message-----
> From: Steve Wise [mailto:swise at opengridcomputing.com]
> Sent: Tuesday, November 27, 2007 4:48 PM
> To: Caitlin Bestler
> Cc: Kanevsky, Arkady; Glenn Grundstrom; Leonid Grossman; openib-
> general at openib.org
> Subject: Re: [ofa-general] Re: iWARP peer-to-peer CM proposal
> 
> Caitlin Bestler wrote:
> > On Nov 27, 2007 3:58 PM, Steve Wise <swise at opengridcomputing.com>
> wrote:
> >
> >> For the short term, I claim we just implement this as part of linux
> >> iwarp connection setup (mandating a 0B read be sent from the active
> >> side).  Your proposal to add meta-data to the private data requires
> a
> >> standards change anyway and is, IMO, the 2nd phase of this whole
> >> enchilada...
> >>
> >> Steve.
> >>
> >
> > I don't see how you can have any solution here that does not require
> meta-data.
> > For non-peer-to-peer connections neither a zero length RDMA Read or
> Write
> > should be sent. An extraneous RDMA Read is particularly onerous for a
> short
> > lived connection that fits the classic active/passive model. So
> *something*
> > is telling the CMA layer that this connection may need an MPA unjam
> action.
> > If that isn't meta-data, what is it?
> 
> I assumed the 0B read would _always_ be sent as part of establishing an
> iWARP connection using linux and the rdma-cm.
> 

That is an extra round-trip per connection setup, which is a significant
penalty for a short lived connection. It is trivial for HPC/peer-to-peer
applications, but would be a killer for something like HTTP over RDMA.

Doing something like this for *every* connection makes it effectively
a change to the MPA protocol. OFA is not the forum for such discussions,
the IETF is.

OFA drafting an understanding of how peer-to-peer applications use the
existing protocol, on the other hand, is quite reasonable. But it has
to be something done by peer-to-peer middleware or by the verbs layer
in response to a flag from the peer-to-peer middleware. Otherwise it
is not augmenting a protocol, it is changing it.