[ofa-general] uDAPL DTO completion question.
Jie.Cai at cs.anu.edu.au
Thu Apr 30 21:35:45 PDT 2009
Thanks for the help along my understanding with IB and uDAPL.
Is that possible to spin remote memory of a rdma atomic compare and swap
I have wrote a program that initiator atomic cmp_swap a value to a
Instead of sending a message to notify the remoter about the completion
the remoter actually doing a memory spin to test the update on the
memory (e.g. while(target == 0);).
However, at remote side, this while loops goes infinitely, and the
initiator has already received DAT_IB_DTO_EVENT.
I don't really understand what's going on, and what would be a correct
way to spin memory for checking remote
Caitlin Bestler wrote:
> On Tue, Mar 31, 2009 at 11:41 PM, Jie Cai <Jie.Cai at cs.anu.edu.au> wrote:
>> Understood now. A further question is here again.
>> To implement software level acknowledgment to inform initiator that data
>> has been available for remoter, is that possible to use a busy loop at
>> side to detect the last element of transferring has appear in the memory.
>> Or remoter has to wait for the event of recv matching initiator's send, then
>> send a message back to initiator as a acknowledgment?
> There are two issues when spinning on a remote memory update.
> The first is that packets may be received and processed out of order,
> especially for iWARP. Therefore the fact that the last byte has been
> received and placed does not guarantee that the prior packets have
> been received and placed.
> More importantly, the order in which updates become visible to a
> specific software thread can make the order of updates unpredictable
> to the application.
> When delivering a completion the Provider is responsible for dealing
> with both of these problems. So when you reap a completion from the
> CQ, the operation it represents (and all prior operations) are complete.
> There are no gaps in received packets, nothing is still sitting on an
> Adapter buffer waiting to be placed in host memory.
> If your application does not want to block you can consider polling
> the cq whether than enabling notifications. But polling memory locations
> directly should only be done when you're willing to have bus/adapter
> specific dependencies. You working code might stop working when
> your network changes, or you install a new Adapter that has a different
> strategy for optimizing its writes over the PCIe bus.
More information about the general