[ofa-general] post_recv question
xma at us.ibm.com
Thu Feb 21 10:09:14 PST 2008
> Here is the timing sequence:
> t0: app calls post_recv
> t1: post_recv code builds a hw-specific WR in the hw work queue
> t2: post_recv code rings a doorbell (write to adapter mem or register)
> t3: post_recv returns
> t4: <app assumes the buffer is ready>
> t5: device HW dma engine moves the WR to adapter memory
> t6: device FW prepares the HW RQ entry making the buffer available.
> Note at time t4, the application thinks its ready, but its really not
> ready until t6.
> This clearly is a implementation-specific issue. But I was under the
> assumption that all the RDMA HW behaves this way. Maybe not?
> To further complicate things, this race condition is never seen _if_ the
> application uses the same QP to advertise (send a credit allowing the
> peer to SEND) the RECV buffer availability. So if the app posts a SEND
> after the RECV is posted and that SEND allows the peer access to the
> RECV buffer, then everything is ok. This is due to the fact that the
> FW/HW will process the SEND only after processing the RECV. If the app
> uses a different QP to post the SEND advertising the RECV, then the race
> condition exists allowing the peer to SEND into that RECV buffer before
> the HW makes it ready.
> This all assumes a specific design of rdma hw. Maybe nobody else has
> this issue?
> Maybe I'm not making sense. :)
I think your descriptions here match what Ralph found RNR in IPoIB-CM.
Does this make sense?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the general