[dat-discussions] [openib-general] [RFC] DAT2.0immediatedataproposal

[Caitlin Bestler]

dat-discussions at yahoogroups.com wrote:
> We have problem no matter which option we choose.
> The current Transport Level Requirement state:
> 
> There is a one-to-one correspondence between send operation
> on one Endpoint of the Connection and recv operations on the
> other Endpoint of the Connection.
> There is no correspondence between RDMA operations on one
> Endpoint of the Connection and recv or send data transfer
> operation on the other Endpoint of the Connection.
> Receive operations on a Connection must be completed in the
> order of posting of their corresponding sends.
> 
> The Immediate data and Atomic ops violate these requirements
> including ordering rules. 
> 
> I had started updating these rules when I generated the first
> draft of the requirements. They are included in the enclosed pdf file.
> But they do not cover Atomic ops that also impact transport
> requirements. This chapter of the spec have not been changed since
> DAPL 1.0 
> and I am very concern with any changes to it.
> 
> Arkady
> 

If "RDMA Write with Immediate" is viewed as being the equivalent
of doing RDMA Write and then an RDMA Send the correspondence
rule is maintained. But *only* if the "rdma write with immediate"
has all of the semantics of a Send.

Atomics do not violate the rules if you view them as being a
variation on an RDMA Read. They are an RDMA Read with modify.
The real question is whether it makes sense to put it in the
RDMA device. It is also not subject to emulation at a highe
layer. 

With send with invalidate we know how InfiniBand *will* support
it, because of the IB 1.2 verbs. We do not know that for atomics
over iWARP. We do not know whether it will be added, more importantly
we do not know *how* it would be added if it were added. That makes
coming up with a transport neutral definition very premature.
In particular, if atomics were added to iWARP there is a distinct
design option where it would *not* be the same work queue as
RDMA Reads (adding atomics through Queue ID 3 would make 
layering on top of a current implementation much easier. But
it would mean that atomic credits would be distinct from read
credits. This is a very strong reason to defer attempting to
define RDMA Atomics in a transport neutral fashion.