[openib-general] dapl broken for iWARP

Fri Feb 9 07:29:57 PST 2007

Mike,
this is not a DAPL issue.
There are 2 ways to deal with it.
One is for all ULPs to use private data to exchange CM info.
yes, some ULPs, like SDP do that in hello world message.

Another is to let CM handle it.
This way ULP does not have to deal with it.
This is analogous to the IBTA CM IP addressing Annex.
It ensure backwards compatibility and does not break any existing apps
which use MPA as specified by IETF.

No need to bother IETF until we have it working.
Thanks,

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300

> -----Original Message-----
> From: Michael Krause [mailto:krause at cup.hp.com] 
> Sent: Thursday, February 08, 2007 4:27 PM
> To: Kanevsky, Arkady; Steve Wise; Arlin Davis
> Cc: openib-general
> Subject: Re: [openib-general] dapl broken for iWARP
> 
> At 07:43 AM 2/8/2007, Kanevsky, Arkady wrote:
> >That is correct.
> >I am working with Krishna on it.
> >Expect patches soon.
> >
> >By the way the problem is not DAPL specific and so is a proposed 
> >solution.
> >
> >There are 3 aspects of the solution.
> >One is APIs. We suggest that we do not augment these.
> >That is a connection requestor sets its QP RDMA ORD and IRD.
> >When connection is established user can check the QP RDMA 
> ORD and IRD 
> >to see what he has now to use over the connection.
> >We may consider to extend QP attributes to support transport 
> specific 
> >parameters passing in the future.
> >For example, iWARP MPA CRC request.
> >
> >Second is the semantic that CM provides.
> >The proposal is to match IBCM semantic.
> >That is CM guarantee that local IRD is >= remote ORD.
> >This guarantees that incoming RDMA Read requests will not 
> overwhelm the 
> >QP RDMA Read capabilities.
> >Again there is not changes to IBCM only to IWCM.
> >Notice that as part of this IWCM will pass down to driver 
> and extract 
> >from driver needed info.
> >
> >The final part is iWARP CM extension to exchange RDMA ORD, IRD.
> >This is similar to IBTA Annex for IP Addressing.
> >The harder part that this will eventually require IETF MPA spec 
> >extension, and the fact that MPA protocol is implemented in 
> RNIC HW by 
> >many vendors, and hence can not be done by IWCM itself.
> 
> We looked at this quite a bit during the creation of the 
> specification.   All of the targeted usage models exchange 
> this information 
> as part of their "hello" or login exchanges.    As such, the 
> "hum" was to 
> not change MPA to communicate such information and leave it 
> to software to 
> exchange these values through existing mechanisms.   I 
> seriously doubt 
> there will be much support for modifying the MPA 
> specification at this stage since the implementations are 
> largely complete and a modification would have to deal with 
> the legacy interoperability issue which likely would be 
> solved in software any way.  It would be simpler to simply 
> modify the underlying DAPL implementation to exchange the 
> information and keep this hidden from both the application 
> and the RNIC providers.
> 
> Mike
> 
> 
> >Thanks,
> >
> >Arkady Kanevsky                       email: arkady at netapp.com
> >Network Appliance Inc.               phone: 781-768-5395
> >1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >Waltham, MA 02451                   central phone: 781-768-5300
> >
> >
> > > -----Original Message-----
> > > From: Steve Wise [mailto:swise at opengridcomputing.com]
> > > Sent: Wednesday, February 07, 2007 6:12 PM
> > > To: Arlin Davis
> > > Cc: openib-general
> > > Subject: Re: [openib-general] dapl broken for iWARP
> > >
> > > On Wed, 2007-02-07 at 15:05 -0800, Arlin Davis wrote:
> > > > Steve Wise wrote:
> > > >
> > > > >On Wed, 2007-02-07 at 14:02 -0600, Steve Wise wrote:
> > > > >
> > > > >
> > > > >>Arlin,
> > > > >>
> > > > >>The OFED dapl code is assuming the responder_resources and
> > > > >>initiator_depth passed up on a connection request event
> > > are from the
> > > > >>remote peer.  This doesn't happen for iWARP.  In the
> > > current iWARP
> > > > >>specifications, its up to the application to exchange this
> > > > >>information somehow. So these are defaulting to 0 on the
> > > server side
> > > > >>of any dapl connection over iWARP.
> > > > >>
> > > > >>This is a fairly recent change, I think.  We need to 
> come up with
> > > > >>some way to deal with this for OFED 1.2 IMO.
> > > > >>
> > > > >>
> > > > Yes, this was changed recently to sync up with the 
> rdma_cm changes
> > > > that exposed the values.
> > > >
> > > > >>
> > > > >>
> > > > >
> > > > >The IWCM could set these to the device max values for instance.
> > > > >
> > > > >
> > > > That would work fine as long as you know the remote
> > > settings will be
> > > > equal or better. The provider just sets the min of 
> local device max
> > > > values and the remote values provided with the request.
> > > >
> > >
> > > I know Krishna Kumar is working on a solution for exchanging
> > > this info in private data so the IWCM can "do the right
> > > thing".  Stay tuned for a patch series to review for this.
> > > But this functionality is definitely post OFED-1.2.
> > >
> > >
> > > So for the OFED-1.2, I will set these to the device max 
> in the IWCM.
> > > Assuming the other side is OFED 1.2 DAPL, then it will work fine.
> > >
> > > Steve.
> > >
> > >
> > >
> > > _______________________________________________
> > > openib-general mailing list
> > > openib-general at openib.org
> > > http://openib.org/mailman/listinfo/openib-general
> > >
> > > To unsubscribe, please visit
> > > http://openib.org/mailman/listinfo/openib-general
> > >
> >
> >_______________________________________________
> >openib-general mailing list
> >openib-general at openib.org
> >http://openib.org/mailman/listinfo/openib-general
> >
> >To unsubscribe, please visit 
> >http://openib.org/mailman/listinfo/openib-general
> 
> 