[ofa-general] Re: iWARP peer-to-peer CM proposal

[Tom Tucker]

On Wed, 2007-11-28 at 11:43 -0500, Caitlin Bestler wrote:
> 
> > -----Original Message-----
> > From: Steve Wise [mailto:swise at opengridcomputing.com]
> > Sent: Tuesday, November 27, 2007 4:48 PM
> > To: Caitlin Bestler
> > Cc: Kanevsky, Arkady; Glenn Grundstrom; Leonid Grossman; openib-
> > general at openib.org
> > Subject: Re: [ofa-general] Re: iWARP peer-to-peer CM proposal
> > 
> > Caitlin Bestler wrote:
> > > On Nov 27, 2007 3:58 PM, Steve Wise <swise at opengridcomputing.com>
> > wrote:
> > >
> > >> For the short term, I claim we just implement this as part of linux
> > >> iwarp connection setup (mandating a 0B read be sent from the active
> > >> side).  Your proposal to add meta-data to the private data requires
> > a
> > >> standards change anyway and is, IMO, the 2nd phase of this whole
> > >> enchilada...
> > >>
> > >> Steve.
> > >>
> > >
> > > I don't see how you can have any solution here that does not require
> > meta-data.
> > > For non-peer-to-peer connections neither a zero length RDMA Read or
> > Write
> > > should be sent. An extraneous RDMA Read is particularly onerous for a
> > short
> > > lived connection that fits the classic active/passive model. So
> > *something*
> > > is telling the CMA layer that this connection may need an MPA unjam
> > action.
> > > If that isn't meta-data, what is it?
> > 
> > I assumed the 0B read would _always_ be sent as part of establishing an
> > iWARP connection using linux and the rdma-cm.
> > 
> 
> That is an extra round-trip per connection setup, which is a significant
> penalty for a short lived connection. It is trivial for HPC/peer-to-peer
> applications, but would be a killer for something like HTTP over RDMA.
> 

I find it hard to get excited about optimizing short lived connections
for RDMA. I simply don't think it's an interesting use case. And btw,
HTTP long ago got rid of short lived connections because it's painful
even on TCP.

> Doing something like this for *every* connection makes it effectively
> a change to the MPA protocol.

Uh. No, it doesn't. Normalizing the behavior of applications during
connection setup doesn't change the underlying protocol. It adds another
one on top.

>  OFA is not the forum for such discussions,
> the IETF is.

My living room, the dinner table, the local bar and this mailing list
are perfectly acceptable forums for discussing a protocol. The IETF is
the forum for standardizing one. Right now, I don't think we're ready to
standardize, because we're still exploring the options; the first of
which is NOT changing MPA.

This group has the unique benefit of actually USING and IMPLEMENTING the
protocol and therefore has some beneficial insights that may and should
be shared. All that said revving the MPA protocol is way down the road. 

> 
> OFA drafting an understanding of how peer-to-peer applications use the
> existing protocol, on the other hand, is quite reasonable. 

That's step 1 and the 0B READ is one way to do it.

> But it has
> to be something done by peer-to-peer middleware or by the verbs layer
> in response to a flag from the peer-to-peer middleware. Otherwise it
> is not augmenting a protocol, it is changing it.
> 

The flag may be useful, however, I don't see the connection between the
flag and complying with the MPA protocol.

> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general