[ofa-general] verbs/hardware question

[Steve Wise]

Doug Ledford wrote:
> On Thu, 2007-10-11 at 12:39 -0500, Steve Wise wrote:
>> Doug Ledford wrote:
>>> So, one of the options when creating a QP is the max inline data size.
>>> If I understand this correctly, for any send up to that size, the
>>> payload of that send will be transmitted to the receiving side along
>>> with the request to send.  
>> What it really means is the payload is DMA'd to the HW on the local side 
>> in the work request itself as opposed being DMA'd down in a 2nd 
>> transaction after the WR is DMA'd and processed.  It has no end-to-end 
>> significance.  Other than to reduce the latency needed to transfer the data.
> 
> OK, that clears things up for me ;-)
> 
>>> This reduces back and forth packet counts on
>>> the wire in the case that the receiving side is good to go, because it
>>> basically just responds with "OK, got it" and you're done.  
>> I don't think this is true.  Definitely not with iWARP.  INLINE is just 
>> an optimization to push small amts of data downto the local adapter as 
>> part of the work request, thus avoiding 2 DMA's.
> 
>> Even though you create the QP with the inline option, only WRs that pass 
>> in the IBV_SEND_INLINE flag will do inline processing, so you can 
>> control this functionality at a per-WR basis.
> 
> Hmm..that raises a question on my part.  You don't call ibv_reg_mr on
> the wr itself, so if the data is pushed with the wr, do you still need
> to call ibv_reg_mr on the data separately?
> 

The WR DMA'd by the HW is actually built in memory that is setup for the 
adapter to DMA from.  Whether that is really done via ibv_reg_mr or some 
other method is provider/vendor specific.  So the WR you pass into 
ibv_post_send() is always copied and munged into the HW-specific memory 
and format.  For inline sends, the data you pass in via the SGL is 
copied into the HW-specific WR memory as well.

And from the man page on ibv_post_send(), I conclude you do _not_ have 
to register the payload memory used in an INLINE send:

>        IBV_SEND_INLINE  Send data in given gather list as inline data
>               in a send WQE.  Valid only for Send and RDMA Write.  The L_Key will not be checked.


Steve.