[ofa-general] post_recv question

[Shirley Ma]

Hello Steve,

> Here is the timing sequence:
>
> t0: app calls post_recv
> t1: post_recv code builds a hw-specific WR in the hw work queue
> t2: post_recv code rings a doorbell (write to adapter mem or register)
> t3: post_recv returns
> t4: <app assumes the buffer is ready>
> t5: device HW dma engine moves the WR to adapter memory
> t6: device FW prepares the HW RQ entry making the buffer available.
>
> Note at time t4, the application thinks its ready, but its really not
> ready until t6.
>
> This clearly is a implementation-specific issue.  But I was under the
> assumption that all the RDMA HW behaves this way.  Maybe not?
>
> To further complicate things, this race condition is never seen _if_ the
> application uses the same QP to advertise (send a credit allowing the
> peer to SEND) the RECV buffer availability.  So if the app posts a SEND
> after the RECV is posted and that SEND allows the peer access to the
> RECV buffer, then everything is ok.  This is due to the fact that the
> FW/HW will process the SEND only after processing the RECV.  If the app
> uses a different QP to post the SEND advertising the RECV, then the race
> condition exists allowing the peer to SEND into that RECV buffer before
> the HW makes it ready.
>
> This all assumes a specific design of rdma hw.  Maybe nobody else has
> this issue?
>
> Maybe I'm not making sense. :)

I think your descriptions here match what Ralph found RNR in IPoIB-CM.

Ralph,

      Does this make sense?

Thanks
Shirley
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080221/9b3e6566/attachment.html>