[ofa-general] Re: [OMPI devel] OMPI over ofed udapl - bugs opened

[Sean Hefty]

> The reason it is hard or impossible to solve this in the DAPL layer is
> that any rdma operation on the QP affects the state of that QP and the
> associate CQs.  In addition, if you use an RDMA send to enforce this you
> impact the other side by consuming a RECV buffer. So its hard if not
> impossible to do this under the covers without affecting the
> application's resources.

I agree that this is hard, but I don't believe that it's impossible.

> Also, the DAPL specification had a goal to not impose any additional
> protocol on the wire.  If you add this under the covers, then you add
> such a "protocol" and break interoperability between a connection
> accessed via DAPL on one end and some other API on the other end.

IMO, this is a unrealized dream.  DAPL does generate wire protocol.  For 
example, when running over IB, DAPL's selection of a service ID and CM protocol 
is visible on the wire.  A DAPL that establishes connections using the RDMA CM 
will likely have a different wire protocol than a version of DAPL that 
establishes connections talking directly to the IB CM.  The two DAPLs will not 
interoperate unless they agree on how they will map to service IDs and, in the 
case of using the RDMA CM, the format of the private data carried in the CM 
messages.

Even in the case of iWarp, DAPL's selection of a local port number affects the 
data visible on the wire.  TO communicate, a remote end point must know how this 
mapping occurs.

- Sean