<html>
<body>
<font size=3>At 11:40 AM 10/11/2007, Steve Wise wrote:<br><br>
<br>
<blockquote type=cite class=cite cite="">Doug Ledford wrote:<br>
<blockquote type=cite class=cite cite="">On Thu, 2007-10-11 at 12:39
-0500, Steve Wise wrote:<br>
<blockquote type=cite class=cite cite="">Doug Ledford wrote:<br>
<blockquote type=cite class=cite cite="">So, one of the options when
creating a QP is the max inline data size.<br>
If I understand this correctly, for any send up to that size, the<br>
payload of that send will be transmitted to the receiving side along<br>
with the request to send. </blockquote>What it really means is the
payload is DMA'd to the HW on the local side in the work request itself
as opposed being DMA'd down in a 2nd transaction after the WR is DMA'd
and processed. </font></blockquote></blockquote></blockquote><br>
Typically it is a series of MMIO writes coalesced and not DMA operations
on the PCI bus. In-line eliminates the latency associated
with a MMIO write to trigger a DMA Read request to be generated by the
device and then the subsequent completion(s) which is then followed by
another DMA Read request and one or more completions.<br><br>
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite=""><font size=3> It has no
end-to-end significance.
</font></blockquote></blockquote></blockquote><br>
Correct. WR + Data in-line is a common technique used in a variety
of I/O solutions for a number of years now. The degree of
performance gains from write coalesce varies by processor / chipset as
well as over time. The in-line itself <br><br>
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite=""><font size=3> Other than to
reduce the latency needed to transfer the data.</blockquote>OK, that
clears things up for me ;-)<br><br>
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite="">This reduces back and forth
packet counts on<br>
the wire in the case that the receiving side is good to go, because
it<br>
basically just responds with "OK, got it" and you're
done. </blockquote>I don't think this is true. Definitely not
with iWARP. INLINE is just an optimization to push small amts of
data downto the local adapter as part of the work request, thus avoiding
2 DMA's.</font></blockquote></blockquote></blockquote><br>
Correct. It is a local only operation. <br><br>
Mike<br><br>
<br>
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite=""><font size=3>Even though you
create the QP with the inline option, only WRs that pass in the
IBV_SEND_INLINE flag will do inline processing, so you can control this
functionality at a per-WR basis.</blockquote>Hmm..that raises a question
on my part. You don't call ibv_reg_mr on<br>
the wr itself, so if the data is pushed with the wr, do you still
need<br>
to call ibv_reg_mr on the data separately?<br>
</blockquote><br>
The WR DMA'd by the HW is actually built in memory that is setup for the
adapter to DMA from. Whether that is really done via ibv_reg_mr or
some other method is provider/vendor specific. So the WR you pass
into ibv_post_send() is always copied and munged into the HW-specific
memory and format. For inline sends, the data you pass in via the
SGL is copied into the HW-specific WR memory as well.<br><br>
And from the man page on ibv_post_send(), I conclude you do _not_ have to
register the payload memory used in an INLINE send:<br><br>
<blockquote type=cite class=cite cite="">
IBV_SEND_INLINE Send data in
given gather list as inline data<br>
in a send WQE. Valid only for Send and RDMA Write. The L_Key
will not be checked.</blockquote><br><br>
Steve.<br>
_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a></font></blockquote>
</body>
</html>