[ofa-general] post_recv question

[Steve Wise]

Eli Cohen wrote:
> When you post to a receive queue, the buffer does not belong the
> application until after it returned to the app. While it belongs to the
> HW, the hardware may use it to write the contents of send messages
> targeting the QP - during this time the driver should no use this
> buffer. When you poll the CQ referenced by the receive queue, you will
> get a CQE with the work request id of the buffer you posted and then you
> know the buffer is back in the driver ownership.
> 

This has nothing to do with my questions.


> On Thu, 2008-02-21 at 09:41 -0600, Steve Wise wrote:
>> Hey all:
>>
>> I have a question regarding exactly _when_ a posted recv buffer is 
>> available for the HW to use:
>>
>> Consider that the post_recv methods usually just program a hw-specific 
>> WR in the RQ, then ring a doorbell, then return.  There is a delta 
>> period between when the app returns from the post_recv call and when the 
>> HW actually DMA's the WR and programs up the HW to enable that buffer. 
>> (I'm assumming a specific HW design here, but I _think_ most HW behaves 
>> this way?).
>>
>> If this is all true, then from the apps point of view, the buffer isn't 
>> really available when it returns from post_recv.  This can lead to 
>> conditions where the app advertises that recv buffer to the peer via 
>> some out of band channel, and the peer posts a SEND which arrives 
>> _before_ the HW has actually setup the RECV buffer.
>>
>> Granted, this hole is small, but does it exist for nes, mthca, ehca, and 
>> ipath libs/drivers?  Or do they _not_ have this issue?
>>
>> Does the IBTA spec discuss this at all?  Most importantly, does the IBTA 
>> spec and/or the iWARP verbs spec _mandate_ that the buffer is actually 
>> available when the post_recv() method returns (I didn't find it in the 
>> iWARP spec)?  If such a mandate exists, then it would force post_recv() 
>> methods to stall and/or somehow know when the HW has completed setting 
>> up the recv buffer.  This would kill performance IMO and I think no such 
>> mandate exists, but I wanted to know what others think.
>>
>> Maybe this isn't an issue with mthca/ehca/ipath/nes?
>>
>>
>> Thanks,
>>
>> Steve.
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general