[ewg] Re: [OMPI devel] OMPI over ofed udapl - bugs opened

[Andrew Friedley]

Steve Wise wrote:
> On Wed, 2007-05-09 at 16:15 -0700, Andrew Friedley wrote:
>> Steve Wise wrote:
>>> There have been a series of discussions on the ofa general list about
>>> this issue, and the conclusion to date is that it cannot be resolved in
>>> the rdma-cm or iwarp-cm code of the linux rdma stack.  Mainly because
>>> sending an RDMA message involves the ULP's work queue and completion
>>> queue, so the CM cannot do this under the covers in a mannor that
>>> doesn't affect the application.  Thus, the applications must deal with
>>> this.
>> Why can't uDAPL deal with this?  As a uDAPL user, I really don't care 
>> what API uDAPL is using under the hood to move data from one place to 
>> another, nor the quirks of that API.  The whole point of uDAPL is to 
>> form a network-agnostic abstraction layer.  AFAIK, the uDAPL spec 
>> doesn't enforce any such requirement on RDMA communication either.  In 
>> my opinion, exposing such behavior above uDAPL is incorrect and is part 
>> of why uDAPL has seen limited adoption -- every single uDAPL 
>> implementation behaves in different ways, making it extremely difficult 
>> to write an application to work on any uDAPL implementation.  Sorry if 
>> this sounds harsh, but this comes from many hours of banging my head on 
>> the wall due to working around these sorts of problems :)
>>
> 
> I understand your frustration.  I think the MPA protocol is deficient in
> this respect and should have required the necessary "first FPDU" to be
> sent under the covers by the RNICs. A RTR packet if you will.  To
> resolve this issue "properly", in my opinion, would involve changing the
> IETF MPA spec and also breaking all the existing iwarp HW.  We can't do
> that.

Understood.

> The reason it is hard or impossible to solve this in the DAPL layer is
> that any rdma operation on the QP affects the state of that QP and the
> associate CQs.  In addition, if you use an RDMA send to enforce this you
> impact the other side by consuming a RECV buffer. So its hard if not
> impossible to do this under the covers without affecting the
> application's resources.

Is there no way to do this before passing connection established events 
to the uDAPL consumer?  I need to go read up on the uDAPL API to really 
understand why this wouldn't work.

> 
> Also, the DAPL specification had a goal to not impose any additional
> protocol on the wire.  If you add this under the covers, then you add
> such a "protocol" and break interoperability between a connection
> accessed via DAPL on one end and some other API on the other end.

So I guess there's no 'right' solution, at least at the uDAPL level. 
With RDMACM/OFA verbs, there's at least the argument that you can design 
the API/semantics however you please, while uDAPL is already standardized.

I hope you guys are documenting this in a way that makes this issue 
extremely clear to both uDAPL and OFA verbs (is this the right naming?) 
users.  Maybe it's been done already, but is it possible to emit some 
sort of loud warning/error when the accept()'ing side tries to send 
before a receive?

Andrew