[ofa-general] uDAPL DTO completion question.

[Jie Cai]

Thanks for the help along my understanding with IB and uDAPL.

Is that possible to spin remote memory of a rdma atomic compare and swap 
(dat_ib_post_cmp_and_swap())?

I have wrote a program that initiator atomic cmp_swap a value to a 
remote memory.

Instead of sending a message to notify the remoter about the completion 
of cmp_swap,
the remoter actually doing a memory spin to test the update on the 
memory (e.g. while(target == 0);).

However, at remote side, this while loops goes infinitely, and the 
initiator has already received DAT_IB_DTO_EVENT.

I don't really understand what's going on, and what would be a correct 
way to spin memory for checking remote
write updates.

Any suggestions?

Regards,
Jie

-- 
Jie Cai



Caitlin Bestler wrote:
> On Tue, Mar 31, 2009 at 11:41 PM, Jie Cai <Jie.Cai at cs.anu.edu.au> wrote:
>   
>> Understood now. A further question is here again.
>>
>> To implement software level acknowledgment to inform initiator that data
>> has been available for remoter, is that possible to use a busy loop at
>> remote
>> side to detect the last element of transferring has appear in the memory.
>>
>> Or remoter has to wait for the event of recv matching initiator's send, then
>> send a message back to initiator as a acknowledgment?
>>
>>     
>
> There are two issues when spinning on a remote memory update.
>
> The first is that packets may be received and processed out of order,
> especially for iWARP. Therefore the fact that the last byte has been
> received and placed does not guarantee that the prior packets have
> been received and placed.
>
> More importantly, the order in which updates become visible to a
> specific software thread can make the order of updates unpredictable
> to the application.
>
> When delivering a completion the Provider is responsible for dealing
> with both of these problems. So when you reap a completion from the
> CQ, the operation it represents (and all prior operations) are complete.
> There are no gaps in received packets, nothing is still sitting on an
> Adapter buffer waiting to be placed in host memory.
>
> If your application does not want to block you can consider polling
> the cq whether than enabling notifications. But polling memory locations
> directly should only be done when you're willing to have bus/adapter
> specific dependencies. You working code might stop working when
> your network changes, or you install a new Adapter that has a different
> strategy for optimizing its writes over the PCIe bus.
>