[ofa-general] post_recv question

[James Lentini]

On Thu, 21 Feb 2008, Steve Wise wrote:

> Hey all:
> 
> I have a question regarding exactly _when_ a posted recv buffer is available
> for the HW to use:
> 
> Consider that the post_recv methods usually just program a hw-specific WR in
> the RQ, then ring a doorbell, then return.  There is a delta period between
> when the app returns from the post_recv call and when the HW actually DMA's
> the WR and programs up the HW to enable that buffer. (I'm assumming a specific
> HW design here, but I _think_ most HW behaves this way?).
> 
> If this is all true, then from the apps point of view, the buffer isn't really
> available when it returns from post_recv.  This can lead to conditions where
> the app advertises that recv buffer to the peer via some out of band channel,
> and the peer posts a SEND which arrives _before_ the HW has actually setup the
> RECV buffer.
> 
> Granted, this hole is small, but does it exist for nes, mthca, ehca, and ipath
> libs/drivers?  Or do they _not_ have this issue?
> 
> Does the IBTA spec discuss this at all?  Most importantly, does the IBTA spec
> and/or the iWARP verbs spec _mandate_ that the buffer is actually available
> when the post_recv() method returns (I didn't find it in the iWARP spec)?  If
> such a mandate exists, then it would force post_recv() methods to stall and/or
> somehow know when the HW has completed setting up the recv buffer.  This would
> kill performance IMO and I think no such mandate exists, but I wanted to know
> what others think.
> 
> Maybe this isn't an issue with mthca/ehca/ipath/nes?
> 
> 
> Thanks,
> 
> Steve.

>From the RDMA application perspective, the application has to assume 
that when post_recv() returns, the RECV WR is on the QP's recv queue 
since there are no APIs for the application to query the 
availability/eligibility of a RECV.