<html><body>
<p><tt>Hello Steve,</tt><br>
<br>
<tt>> Here is the timing sequence:<br>
> <br>
> t0: app calls post_recv<br>
> t1: post_recv code builds a hw-specific WR in the hw work queue<br>
> t2: post_recv code rings a doorbell (write to adapter mem or register)<br>
> t3: post_recv returns<br>
> t4: <app assumes the buffer is ready><br>
> t5: device HW dma engine moves the WR to adapter memory<br>
> t6: device FW prepares the HW RQ entry making the buffer available.<br>
> <br>
> Note at time t4, the application thinks its ready, but its really not <br>
> ready until t6.<br>
> <br>
> This clearly is a implementation-specific issue. But I was under the <br>
> assumption that all the RDMA HW behaves this way. Maybe not?<br>
> <br>
> To further complicate things, this race condition is never seen _if_ the <br>
> application uses the same QP to advertise (send a credit allowing the <br>
> peer to SEND) the RECV buffer availability. So if the app posts a SEND <br>
> after the RECV is posted and that SEND allows the peer access to the <br>
> RECV buffer, then everything is ok. This is due to the fact that the <br>
> FW/HW will process the SEND only after processing the RECV. If the app <br>
> uses a different QP to post the SEND advertising the RECV, then the race <br>
> condition exists allowing the peer to SEND into that RECV buffer before <br>
> the HW makes it ready.<br>
> <br>
> This all assumes a specific design of rdma hw. Maybe nobody else has <br>
> this issue?<br>
> <br>
> Maybe I'm not making sense. :)<br>
</tt><br>
<tt>I think your descriptions here match what Ralph found RNR in IPoIB-CM.</tt><br>
<br>
<tt>Ralph,</tt><br>
<br>
<tt> Does this make sense?</tt><br>
<br>
<tt>Thanks</tt><br>
<tt>Shirley</tt></body></html>