[ofa-general] Re: iWARP peer-to-peer CM proposal

[Felix Marti]

> -----Original Message-----
> From: general-bounces at lists.openfabrics.org [mailto:general-
> bounces at lists.openfabrics.org] On Behalf Of Kanevsky, Arkady
> Sent: Wednesday, November 28, 2007 5:30 AM
> To: Steve Wise; Caitlin Bestler
> Cc: Leonid Grossman; openib-general at openib.org
> Subject: RE: [ofa-general] Re: iWARP peer-to-peer CM proposal
> 
> Agree with initiator/client sending signalled 0B RDMA Read.
> This will handle client side.
> 
> Still not 100% clear on passive/server side.
> Two issues which bothers me.
> 1. Is "bogus" S-tag allowed for incomming RDMA ops?
> I do not recall that RDDP requires that length is checked before
> S-tag.
> 
> 2. How is "verb" layer on server side knows that RDMA Read op
> came and was done? Is it some back door to vendor FW?
> Will this be kicked for all incoming RDMA Read ops?

As you point out, the server Verbs layer is not aware of an incoming 0B
RDMA Read (or Write for that matter). Hence some kind of magic must
happen in the adapter where we vendors will have a choice: a) just
'unjam' the SQ in the adapter (which means that the CM layer works as
today and the server can post SQ ops before the 'unjam' is received but
they won't make it to the wire) or b) send a back-door command to the CM
which can then move the state machine to established only after the
'unjam' is received.

Whatever is done, it cannot happen for all zero-length RDMA Read (or
Write for that matter). Hence the adapter must be informed that that the
next zero-length is the 'unjam' message (which also means that the
server side could, in theory, omit sending the RDMA Read Response,
because the RDMA Read Request was really a 'unjam'... not that I would
be pushing for such an 'optimization' to avoid an extra wire message).

> 
> Arkady Kanevsky                       email: arkady at netapp.com
> Network Appliance Inc.               phone: 781-768-5395
> 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> Waltham, MA 02451                   central phone: 781-768-5300
> 
> 
> > -----Original Message-----
> > From: Steve Wise [mailto:swise at opengridcomputing.com]
> > Sent: Tuesday, November 27, 2007 7:48 PM
> > To: Caitlin Bestler
> > Cc: Kanevsky, Arkady; Glenn Grundstrom; Leonid Grossman;
> > openib-general at openib.org
> > Subject: Re: [ofa-general] Re: iWARP peer-to-peer CM proposal
> >
> > Caitlin Bestler wrote:
> > > On Nov 27, 2007 3:58 PM, Steve Wise
> > <swise at opengridcomputing.com> wrote:
> > >
> > >> For the short term, I claim we just implement this as part
> > of linux
> > >> iwarp connection setup (mandating a 0B read be sent from
> > the active
> > >> side).  Your proposal to add meta-data to the private data
> > requires a
> > >> standards change anyway and is, IMO, the 2nd phase of this whole
> > >> enchilada...
> > >>
> > >> Steve.
> > >>
> > >
> > > I don't see how you can have any solution here that does
> > not require meta-data.
> > > For non-peer-to-peer connections neither a zero length RDMA Read
or
> > > Write should be sent. An extraneous RDMA Read is
> > particularly onerous
> > > for a short lived connection that fits the classic active/passive
> > > model. So *something* is telling the CMA layer that this
> > connection may need an MPA unjam action.
> > > If that isn't meta-data, what is it?
> >
> > I assumed the 0B read would _always_ be sent as part of
> > establishing an iWARP connection using linux and the rdma-cm.
> >
> > >
> > > Further, the RDMA Read solution is adequate whenever the RDMA
Write
> > > solution would have been (although at an unnecessary extra
> > cost), but
> > > as near as I can determine it is not a complete solution. If the
> > > passive side needs an untagged message completion then *something*
> > > needs to send it. How can the CM layer (or, I suppose, the
> > ULP itself)
> > > know that this untagged NOP message must be sent without
meta-data?
> >
> > I believe at Reno we had the current rnic vendors all saying
> > a SEND or 0B read will work.  So:  If someone has current
> > iwarp HW that will _not_
> >   handle this problem by doing the 0B read hack, please speak up
now.
> >
> > >
> > > As I see it, if we want to do the minimum that is required, but be
> > > certain that it is adequate, we need a per-connection setup
> > meta-data exchange.
> >
> > Are you going to prototype this?
> >
> >
> > Steve.
> >
> >
> >
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general