[ofa-general] post_recv question

[Eli Cohen]

When you post to a receive queue, the buffer does not belong the
application until after it returned to the app. While it belongs to the
HW, the hardware may use it to write the contents of send messages
targeting the QP - during this time the driver should no use this
buffer. When you poll the CQ referenced by the receive queue, you will
get a CQE with the work request id of the buffer you posted and then you
know the buffer is back in the driver ownership.

On Thu, 2008-02-21 at 09:41 -0600, Steve Wise wrote:
> Hey all:
> 
> I have a question regarding exactly _when_ a posted recv buffer is 
> available for the HW to use:
> 
> Consider that the post_recv methods usually just program a hw-specific 
> WR in the RQ, then ring a doorbell, then return.  There is a delta 
> period between when the app returns from the post_recv call and when the 
> HW actually DMA's the WR and programs up the HW to enable that buffer. 
> (I'm assumming a specific HW design here, but I _think_ most HW behaves 
> this way?).
> 
> If this is all true, then from the apps point of view, the buffer isn't 
> really available when it returns from post_recv.  This can lead to 
> conditions where the app advertises that recv buffer to the peer via 
> some out of band channel, and the peer posts a SEND which arrives 
> _before_ the HW has actually setup the RECV buffer.
> 
> Granted, this hole is small, but does it exist for nes, mthca, ehca, and 
> ipath libs/drivers?  Or do they _not_ have this issue?
> 
> Does the IBTA spec discuss this at all?  Most importantly, does the IBTA 
> spec and/or the iWARP verbs spec _mandate_ that the buffer is actually 
> available when the post_recv() method returns (I didn't find it in the 
> iWARP spec)?  If such a mandate exists, then it would force post_recv() 
> methods to stall and/or somehow know when the HW has completed setting 
> up the recv buffer.  This would kill performance IMO and I think no such 
> mandate exists, but I wanted to know what others think.
> 
> Maybe this isn't an issue with mthca/ehca/ipath/nes?
> 
> 
> Thanks,
> 
> Steve.
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general