***SPAM*** Re: [ofa-general] uDAPL DTO completion question.

[Caitlin Bestler]

On Tue, Mar 31, 2009 at 11:41 PM, Jie Cai <Jie.Cai at cs.anu.edu.au> wrote:
> Understood now. A further question is here again.
>
> To implement software level acknowledgment to inform initiator that data
> has been available for remoter, is that possible to use a busy loop at
> remote
> side to detect the last element of transferring has appear in the memory.
>
> Or remoter has to wait for the event of recv matching initiator's send, then
> send a message back to initiator as a acknowledgment?
>

There are two issues when spinning on a remote memory update.

The first is that packets may be received and processed out of order,
especially for iWARP. Therefore the fact that the last byte has been
received and placed does not guarantee that the prior packets have
been received and placed.

More importantly, the order in which updates become visible to a
specific software thread can make the order of updates unpredictable
to the application.

When delivering a completion the Provider is responsible for dealing
with both of these problems. So when you reap a completion from the
CQ, the operation it represents (and all prior operations) are complete.
There are no gaps in received packets, nothing is still sitting on an
Adapter buffer waiting to be placed in host memory.

If your application does not want to block you can consider polling
the cq whether than enabling notifications. But polling memory locations
directly should only be done when you're willing to have bus/adapter
specific dependencies. You working code might stop working when
your network changes, or you install a new Adapter that has a different
strategy for optimizing its writes over the PCIe bus.