<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:470757632;
mso-list-type:hybrid;
mso-list-template-ids:-1043036720 1581178398 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l1
{mso-list-id:606501852;
mso-list-type:hybrid;
mso-list-template-ids:-1123512542 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hi Leo,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Sorry for the delay in getting back to you. I think using
inline data makes sense. A couple things though, and it’s hard to
comment inline due to the HTML format so here they are at the top:<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Max inline data is not an in/out parameter to the
INDConnector::CreateEndpoint method. I don’t know if it makes sense
to have it be an input parameter. Aren’t the proper tuning points
dependent on the HCA, rather than the app?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Assuming that the tuning points are specific to the HCA models,
does it make sense to always allocate 400 bytes? Is it always faster to
inline 400 bytes than to DMA the data for all HCAs (InfiniHost 3 LX, EX,
ConnectX, etc?) It seems to me that having the inline data controlled by
the HCA driver rather than the ND provider would make more sense, and allow the
HCA driver to optimize the sweet spot.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>This all assumes that the sweet spot is largely independent of
the host system (CPU, memory bus, PCI bus), and a function of the HCA’s
ability to process requests.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>When you query the QP to get the actual inline data size, you
store the value temporarily in a local variable, then store it in the member
variable, as well as return. Why not just store it in the member variable
and eliminate the local variable?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Ndendpoint.h @ line 176 seems to have whitespace issues.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Lastly, the latency improvement you quote is for RDMA writes
(the nd<b>r</b>pingpong test). You also provide results for ib_send_lat
compared to ndpingpong, but those results aren’t comparable to begin with
because of test methodology (e.g. ib_send_lat reports median latency as typical,
which helps ignore outliers, while ndpingpong calculates average.) Do you
have before/after results for ndpingpong?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Fab<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org] <b>On
Behalf Of </b>Leonid Keller<br>
<b>Sent:</b> Monday, July 27, 2009 1:27 AM<br>
<b>To:</b> Fab Tillier; Tzachi Dar<br>
<b>Cc:</b> ofw@lists.openfabrics.org<br>
<b>Subject:</b> [ofw][patch][ND provider] Improving latency of ms-mpi<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>This patch adds usage of INLINE DATA facility of Mellanox HCAs for
improving latency of ND provider.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>E.g., on our Nehalem computer (Processor Intel(R) Core(TM) i7
CPU 920 @ 2.67GHz, 2660 Mhz, 4 Core(s), 8 Logical Processor(s)) </span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><strong><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>it improved latency from 2.23 us to 1.12 us.</span></strong><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Here are the ideas of the patch:</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'> - by default, ND provider will create QP with
inline data of 400 bytes;</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'> (this can enlarge user's QP size)</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'> - one can change this default by defining env
variable IBNDPROV_MAX_INLINE_SIZE;</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'> - an ND application, while creating QP, can
define the necessary INLINE DATA size; this value takes precedence over the
default one.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Index: ulp/nd/user/NdEndpoint.cpp<br>
===================================================================<br>
--- ulp/nd/user/NdEndpoint.cpp (revision 2310)<br>
+++ ulp/nd/user/NdEndpoint.cpp (working copy)<br>
@@ -41,6 +41,8 @@<br>
#pragma warning( pop )<br>
#include "nddebug.h"<br>
<br>
+extern uint32_t g_nd_max_inline_size;<br>
+<br>
#if defined(EVENT_TRACING)<br>
#ifdef offsetof<br>
#undef offsetof<br>
@@ -96,7 +98,7 @@<br>
__in SIZE_T nOutboundSge,<br>
__in SIZE_T InboundReadLimit,<br>
__in SIZE_T OutboundReadLimit,<br>
- __out_opt SIZE_T* pMaxInlineData<br>
+ __in_opt __out_opt SIZE_T* pMaxInlineData<br>
)<br>
{<br>
ND_ENTER( ND_DBG_NDI );<br>
@@ -117,10 +119,17 @@<br>
m_pParent->m_Ifc.user_verbs.nd_get_qp_state != NULL ||<br>
m_pParent->m_Ifc.user_verbs.pre_destroy_qp != NULL ||<br>
m_pParent->m_Ifc.user_verbs.post_destroy_qp != NULL ||<br>
+
m_pParent->m_Ifc.user_verbs.post_query_qp != NULL ||<br>
m_pParent->m_Ifc.user_verbs.post_send != NULL ||<br>
m_pParent->m_Ifc.user_verbs.post_recv != NULL /*||<br>
m_pParent->m_Ifc.user_verbs.bind_mw != NULL*/ );<br>
<br>
+ UINT32 InlineSize;<br>
+ if ( pMaxInlineData )<br>
+ InlineSize = (UINT32)*pMaxInlineData;<br>
+ else<br>
+ InlineSize = g_nd_max_inline_size;<br>
+ <br>
HRESULT hr = CreateQp(<br>
pInboundCq,<br>
pOutboundCq,<br>
@@ -129,13 +138,25 @@<br>
nInboundSge,<br>
nOutboundSge,<br>
InboundReadLimit,<br>
- OutboundReadLimit );<br>
+ OutboundReadLimit,<br>
+ InlineSize );<br>
<br>
if( FAILED( hr ) )<br>
return hr;<br>
<br>
+ ib_qp_attr_t qp_attr;<br>
+ hr = QueryQp(&qp_attr);<br>
+ if( FAILED( hr ) ) {<br>
+ DestroyQp();<br>
+ return hr;<br>
+ }<br>
+ else<br>
+ InlineSize = (UINT32)qp_attr.sq_max_inline;<br>
+ <br>
+ <br>
m_Ird = (UINT8)InboundReadLimit;<br>
m_Ord = (UINT8)OutboundReadLimit;<br>
+ m_MaxInlineSize = InlineSize;<br>
<br>
// Move the QP to the INIT state so users can post
receives.<br>
hr = ModifyQp( IB_QPS_INIT );<br>
@@ -143,10 +164,7 @@<br>
DestroyQp();<br>
<br>
if( SUCCEEDED( hr ) && pMaxInlineData != NULL
)<br>
- {<br>
- // Worst case.<br>
- *pMaxInlineData = nOutboundSge *
12;<br>
- }<br>
+ *pMaxInlineData = InlineSize;<br>
<br>
return hr;<br>
}<br>
@@ -286,7 +304,11 @@<br>
wr.p_next = NULL;<br>
wr.wr_id = (ULONG_PTR)pResult;<br>
wr.wr_type = WR_SEND;<br>
- wr.send_opt = 0;<br>
+ if ( pResult->BytesTransferred <= m_MaxInlineSize )<br>
+ wr.send_opt = IB_SEND_OPT_INLINE; <br>
+ else<br>
+ wr.send_opt = 0;<br>
+ <br>
if( !(Flags & ND_OP_FLAG_SILENT_SUCCESS) )<br>
wr.send_opt |=
IB_SEND_OPT_SIGNALED;<br>
if( Flags & ND_OP_FLAG_READ_FENCE )<br>
@@ -374,11 +396,15 @@<br>
wr.p_next = NULL;<br>
wr.wr_id = (ULONG_PTR)pResult;<br>
wr.wr_type = WR_SEND;<br>
+ if ( pResult->BytesTransferred <= m_MaxInlineSize )<br>
+ wr.send_opt = IB_SEND_OPT_INLINE; <br>
+ else<br>
+ wr.send_opt = 0;<br>
// We simulate invalidate operations (since we
simulate MW use). We<br>
// put the RKey in the immediate data, the recipient
will do the<br>
// lookup of the MW based on that (as they would with
a regular<br>
// invalidate request).<br>
- wr.send_opt = IB_SEND_OPT_IMMEDIATE;<br>
+ wr.send_opt |= IB_SEND_OPT_IMMEDIATE;<br>
if( !(Flags & ND_OP_FLAG_SILENT_SUCCESS) )<br>
wr.send_opt |=
IB_SEND_OPT_SIGNALED;<br>
if( Flags & ND_OP_FLAG_READ_FENCE )<br>
@@ -665,7 +691,10 @@<br>
wr.p_next = NULL;<br>
wr.wr_id = (ULONG_PTR)pResult;<br>
wr.wr_type = Type;<br>
- wr.send_opt = 0;<br>
+ if ( pResult->BytesTransferred <= m_MaxInlineSize )<br>
+ wr.send_opt = IB_SEND_OPT_INLINE; <br>
+ else<br>
+ wr.send_opt = 0;<br>
if( !(Flags & ND_OP_FLAG_SILENT_SUCCESS) )<br>
wr.send_opt |=
IB_SEND_OPT_SIGNALED;<br>
if( Flags & ND_OP_FLAG_READ_FENCE )<br>
@@ -737,11 +766,14 @@<br>
__in SIZE_T nInboundSge,<br>
__in SIZE_T nOutboundSge,<br>
__in SIZE_T InboundReadLimit,<br>
- __in SIZE_T OutboundReadLimit<br>
+ __in SIZE_T OutboundReadLimit,<br>
+ __in SIZE_T MaxInlineData<br>
)<br>
{<br>
ND_ENTER( ND_DBG_NDI );<br>
<br>
+ if( MaxInlineData > UINT_MAX )<br>
+ return ND_INVALID_PARAMETER_3;<br>
if( nInboundEntries > UINT_MAX )<br>
return ND_INVALID_PARAMETER_4;<br>
if( nOutboundEntries > UINT_MAX )<br>
@@ -764,6 +796,7 @@<br>
qp_ioctl.in.qp_create.rq_depth =
(uint32_t)nInboundEntries;<br>
qp_ioctl.in.qp_create.sq_sge = (uint32_t)nOutboundSge;<br>
qp_ioctl.in.qp_create.rq_sge = (uint32_t)nInboundSge;<br>
+ qp_ioctl.in.qp_create.sq_max_inline =
(uint32_t)MaxInlineData;<br>
qp_ioctl.in.qp_create.h_srq = NULL;<br>
qp_ioctl.in.qp_create.sq_signaled = FALSE;<br>
<br>
@@ -941,4 +974,67 @@<br>
return S_OK;<br>
}<br>
<br>
+HRESULT CEndpoint::QueryQp(<br>
+ __out ib_qp_attr_t *qp_attr<br>
+ )<br>
+{<br>
+ ib_api_status_t status;<br>
+ <br>
+ ND_ENTER( ND_DBG_NDI );<br>
+<br>
+ ual_query_qp_ioctl_t qp_ioctl;<br>
+ cl_memclr( &qp_ioctl, sizeof(qp_ioctl) );<br>
+ qp_ioctl.in.h_qp = m_hQp;<br>
+<br>
+ /* Call the uvp pre call if the vendor library provided a valid ca
handle */<br>
+ if( m_pParent->m_Ifc.user_verbs.pre_query_qp )<br>
+ {<br>
+ /* Pre call to the UVP library */<br>
+ status = m_pParent->m_Ifc.user_verbs.pre_query_qp( m_uQp,
&qp_ioctl.in.umv_buf );<br>
+ if( status != IB_SUCCESS )<br>
+ goto done;<br>
+ }<br>
+<br>
+ DWORD bytes_ret;<br>
+ BOOL fSuccess = DeviceIoControl(<br>
+ m_pParent->m_hSync,<br>
+ UAL_QUERY_QP,<br>
+ &qp_ioctl.in,<br>
+ sizeof(qp_ioctl.in),<br>
+ &qp_ioctl.out,<br>
+ sizeof(qp_ioctl.out),<br>
+ &bytes_ret,<br>
+ NULL<br>
+ );<br>
+<br>
+ if( fSuccess != TRUE || bytes_ret != sizeof(qp_ioctl.out) )<br>
+ status = IB_ERROR;<br>
+ else<br>
+ status = qp_ioctl.out.status;<br>
+<br>
+ /* Call vendor's post_query_qp */<br>
+ CL_ASSERT( m_pParent->m_Ifc.user_verbs.post_query_qp );<br>
+ if( m_pParent->m_Ifc.user_verbs.post_query_qp )<br>
+ {<br>
+ m_pParent->m_Ifc.user_verbs.post_query_qp( m_uQp, status,<br>
+ &qp_ioctl.out.attr, &qp_ioctl.out.umv_buf );<br>
+ }<br>
+<br>
+done:<br>
+ ND_PRINT( TRACE_LEVEL_INFORMATION, ND_DBG_NDI,<br>
+ ("Queried QP %#I64x, QPn %#x, pd %#I64x, context %p, status
%#x \n", <br>
+ m_hQp, m_Qpn, m_pParent->m_hPd, this, status ) );<br>
+<br>
+ switch( status )<br>
+ {<br>
+ case IB_SUCCESS:<br>
+ *qp_attr = qp_ioctl.out.attr;<br>
+ return S_OK;<br>
+<br>
+ default:<br>
+ return ND_UNSUCCESSFUL;<br>
+ }<br>
+ <br>
+}<br>
+<br>
} // namespace<br>
Index: ulp/nd/user/NdEndpoint.h<br>
===================================================================<br>
--- ulp/nd/user/NdEndpoint.h (revision 2310)<br>
+++ ulp/nd/user/NdEndpoint.h (working copy)<br>
@@ -67,7 +67,7 @@<br>
__in SIZE_T nOutboundSge,<br>
__in SIZE_T InboundReadLimit,<br>
__in SIZE_T OutboundReadLimit,<br>
- __out_opt SIZE_T* pMaxInlineData<br>
+ __in_opt __out_opt SIZE_T*
pMaxInlineData<br>
);<br>
<br>
public:<br>
@@ -176,11 +176,16 @@<br>
__in SIZE_T nInboundSge,<br>
__in SIZE_T nOutboundSge,<br>
__in SIZE_T InboundReadLimit,<br>
- __in SIZE_T OutboundReadLimit<br>
+ __in SIZE_T OutboundReadLimit,<br>
+ __in SIZE_T MaxInlineData<br>
);<br>
<br>
void DestroyQp();<br>
<br>
+ HRESULT QueryQp(<br>
+ __out ib_qp_attr_t *qp_attr<br>
+ );<br>
+<br>
HRESULT ModifyQp(<br>
__in ib_qp_state_t NewState<br>
);<br>
@@ -197,6 +202,7 @@<br>
<br>
UINT8 m_Ird;<br>
UINT8 m_Ord;<br>
+ UINT32 m_MaxInlineSize;<br>
};<br>
<br>
} // namespace<br>
Index: ulp/nd/user/NdProv.cpp<br>
===================================================================<br>
--- ulp/nd/user/NdProv.cpp (revision 2310)<br>
+++ ulp/nd/user/NdProv.cpp (working copy)<br>
@@ -57,6 +57,7 @@<br>
uint32_t g_nd_dbg_level = TRACE_LEVEL_ERROR;<br>
/* WPP doesn't want here literals! */<br>
uint32_t g_nd_dbg_flags = 0x80000001; /* ND_DBG_ERROR | ND_DBG_NDI; */<br>
+uint32_t g_nd_max_inline_size = 400;<br>
<br>
HANDLE ghHeap;<br>
<br>
@@ -462,6 +463,8 @@<br>
switch( dwReason )<br>
{<br>
case DLL_PROCESS_ATTACH:<br>
+ TCHAR
env_var[16];<br>
+ DWORD i;<br>
<br>
<br>
#if defined(EVENT_TRACING)<br>
@@ -471,9 +474,6 @@<br>
WPP_INIT_TRACING(L"ibndprov.dll");<br>
#endif<br>
#elif DBG <br>
- TCHAR
env_var[16];<br>
- DWORD i;<br>
-<br>
i = GetEnvironmentVariable(
"IBNDPROV_DBG_LEVEL", env_var, sizeof(env_var) );<br>
if( i && i <= 16 )<br>
{<br>
@@ -494,6 +494,12 @@<br>
GetCurrentProcessId(), g_nd_dbg_level ,g_nd_dbg_flags) );<br>
#endif<br>
<br>
+ i = GetEnvironmentVariable( "IBNDPROV_MAX_INLINE_SIZE",
env_var, sizeof(env_var) );<br>
+ if( i && i <= 16 )<br>
+ {<br>
+ g_nd_max_inline_size = _tcstoul( env_var, NULL, 16 );<br>
+ }<br>
+<br>
ghHeap = HeapCreate( 0, 0, 0
);<br>
if( ghHeap == NULL )<br>
{</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" align=center>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Tzachi Dar <br>
<b>Sent:</b> Monday, July 20, 2009 6:13 PM<br>
<b>To:</b> Leonid Keller<br>
<b>Subject:</b> FW: Improving latency of ms-mpi</span><o:p></o:p></p>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" align=center>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Tzachi Dar <br>
<b>Sent:</b> Tuesday, July 14, 2009 6:39 PM<br>
<b>To:</b> Fab Tillier<br>
<b>Cc:</b> Ishai Rabinovitz; Gilad Shainer; Leonid Keller; Eric Lantz (HPC
GOFAST)<br>
<b>Subject:</b> Improving latency of ms-mpi</span><o:p></o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hi
Fab,</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>I have made some tests on my Nehalem computer
(Processor Intel(R) Core(TM) i7
CPU 920 @ 2.67GHz, 2660
Mhz, 4 Core(s), 8 Logical Processor(s))</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>It seems that the latency that I got for RDMA operations using </span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>ib_write_lat</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'> </span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>is 1.26us</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>On the same machine ndrpingpong.exe gave</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'> 2.23u</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'> which is
almost 1us bigger )-: </span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>After some debugging of the system, it seems that the problem comes
from not using inline send. Using Inline send has reduced the ndrpingpong
latency to </span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:red'>1.12us</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'> Which as far as I know is around 200ns bigger than the Clovertown
numbers.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>[By the way, it seems that the nd pingpong test is better than ours
due to a lazy poll cq, but this is a different issue].</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Since nothing in the nd provider is using inline send I guess that
we can improve all the results on MS-MPI in about 1us by using inline
send.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>I have created a very simple patch that demonstrates what has to be
done in order to allow inline to work. Please note that the patch is over
simplified and will always use inline RDMA operations even when it is not
allowed to.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>The questions that I see are:</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>1) How do we pass the maxinline parameter for creating the qp? Do
we want to add a new parameter, or do we want to use sge * 12 number, or do we
want a big default?</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>2) How do we decide when to send using inline? Do we want to add a
new ND_OP_FLAG, do we want to always send when we can (that is message <
max_inline)?</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Here is the patch created:</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Index: NdEndpoint.cpp<br>
===================================================================<br>
--- NdEndpoint.cpp (revision 4569)<br>
+++ NdEndpoint.cpp (working copy)<br>
@@ -635,7 +635,7 @@<br>
wr.p_next = NULL;<br>
wr.wr_id = (ULONG_PTR)pResult;<br>
wr.wr_type = Type;<br>
- wr.send_opt = 0;<br>
+ wr.send_opt = IB_SEND_OPT_INLINE; //????<br>
if( !(Flags & ND_OP_FLAG_SILENT_SUCCESS) )<br>
wr.send_opt |=
IB_SEND_OPT_SIGNALED;<br>
if( Flags & ND_OP_FLAG_READ_FENCE )<br>
@@ -737,6 +737,8 @@<br>
qp_ioctl.in.qp_create.h_srq = NULL;<br>
qp_ioctl.in.qp_create.sq_signaled = FALSE;<br>
<br>
+ qp_ioctl.in.qp_create.sq_max_inline = 400;<br>
+<br>
/* Pre call to the UVP library */<br>
CL_ASSERT(
m_pParent->m_Ifc.user_verbs.pre_create_qp );<br>
qp_ioctl.in.qp_create.h_sq_cq = pOutboundCq->m_uCq;</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Thanks</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Tzachi</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Here are the more detailed results:</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>C:\Users\herod></span><span style='font-size:10.0pt;font-family:
"Arial","sans-serif";color:red'>ib_write_lat</span><span style='font-size:10.0pt;
font-family:"Arial","sans-serif";color:blue'> -a 11.4.12.128<br>
------------------------------------------------------------------<br>
RDMA_Write Latency Test<br>
Connection type : RC<br>
max inline size 28<br>
local address: LID 0x200, QPN 0x4b000200, PSN 0xe7520000, RKey
0x140004 VAddr 0x00000001630040<br>
remote address: LID 0x100, QPN 0x4a001e00, PSN 0xf6040000, RKey 0x130040
VAddr 0x00000001530040<br>
Mtu : 2048<br>
------------------------------------------------------------------<br>
#bytes #iterations t_min[usec]
t_max[usec] t_typical[usec]<br>
2
1000
0.98
4.89 </span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>1.26</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><br>
4
1000
1.12
1.26
1.26<br>
8
1000
0.98
1.40
1.26<br>
16
1000
0.84 31.71
1.26</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:red'>C:\Users\herod>ib_read_lat -a 11.4.12.128<br>
</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>------------------------------------------------------------------<br>
RDMA_Read Latency Test<br>
max inline size 28<br>
local address: LID 0x200, QPN 0x4a000600, PSN 0xf1560000, RKey
0x13000c VAddr 0x00000001550040<br>
remote address: LID 0x100, QPN 0x4a000400, PSN 0x42390000, RKey 0x13000c
VAddr 0x00000001390040<br>
Mtu : 2048<br>
------------------------------------------------------------------<br>
#bytes #iterations t_min[usec]
t_max[usec] t_typical[usec]<br>
2
1000
1.40
8.94 </span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>2.23</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><br>
4
1000
1.40
2.79
2.23<br>
8
1000
1.96
2.51
2.23<br>
16
1000 1.68
2.79
2.23<br>
32
1000
1.40
9.22
2.23<br>
64
1000
1.40
3.07
2.23<br>
128
1000
2.23
2.79
2.23<br>
256
1000
1.96
2.79
2.51<br>
512
1000
1.96
3.07
2.79<br>
1024
1000
2.51
3.63
3.07<br>
2048
1000
3.07
32.69
3.91<br>
4096
1000
3.91
5.31
4.75<br>
8192
1000
5.87
6.70
6.15<br>
16384
1000
5.31
26.82
8.66<br>
32768
1000
12.29
15.37 14.25<br>
65536
1000
22.63
42.46 24.86<br>
131072
1000
43.30
88.56 46.10<br>
262144
1000
84.09
152.53 88.84<br>
524288
1000
169.02 236.90
174.32<br>
1048576
1000
342.22
382.45 345.02<br>
2097152
1000
653.99
706.51 686.96<br>
4194304
1000
1364.70
1408.00 1370.57<br>
8388608
1000
2736.10
2765.71 2738.62<br>
------------------------------------------------------------------</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>C:\Users\herod></span><span style='font-size:10.0pt;font-family:
"Arial","sans-serif";color:red'>ib_Send_lat -a 11.4.12.128</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><br>
------------------------------------------------------------------<br>
Send Latency Test<br>
Inline data is used up to 400 bytes message<br>
Connection type : RC<br>
test<br>
local address: LID 0x200, QPN 0x4a000800, PSN 0x7a600000, RKey
0x130010 VAddr 0x00000001510040<br>
remote address: LID 0x100, QPN 0x4a000600, PSN 0x6c6e0000, RKey 0x130010
VAddr 0x00000001570040<br>
Mtu : 2048<br>
------------------------------------------------------------------<br>
#bytes #iterations t_min[usec]
t_max[usec] t_typical[usec]<br>
2
1000
1.26
20.95 </span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>1.40</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><br>
4
1000 1.26
5.17
1.40<br>
8
1000
1.12
5.03
1.40</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>C:\Users\herod>q:\projinf4\trunk\bin\user\objfre_wlh_amd64\amd64\</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>ndpingpong.exe</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'> c
11.4.12.128 5000 p<br>
1<br>
Using 8 processors. Sender Frequency is 3579545<br>
1
100000 </span><span style='font-size:10.0pt;
font-family:"Arial","sans-serif";color:red'>3.07</span><span style='font-size:
10.0pt;font-family:"Arial","sans-serif";color:blue'>
106.77 325414<br>
2
100000 3.07
106.77 650829<br>
4
100000 3.07 106.77
1301659<br>
8
100000 3.07 109.31
2603319<br>
16
100000 3.07 106.77
5206638<br>
32
100000 3.07 109.31
10413276</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>C:\Users\herod>q:\projinf4\trunk\bin\user\objfre_wlh_amd64\amd64\</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:red'>ndrpingpong.exe</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'> c
11.4.12.128 5000<br>
p1<br>
Using 8 processors. Sender Frequency is 3579545<br>
1
100000 </span><span style='font-size:10.0pt;
font-family:"Arial","sans-serif";color:red'> 2.23</span><span style='font-size:
10.0pt;font-family:"Arial","sans-serif";color:blue'>
101.39 447627<br>
2
100000 2.23
101.39 895255<br>
4
100000 2.23 101.39
1790510<br>
8
100000 2.23 101.39
3581020<br>
16
100000 2.23 101.39
7162041<br>
32
100000 2.23 108.37
14324082<br>
64
100000 2.23 104.88
28648164<br>
128
100000 2.23 111.91
57296329<br>
256
100000 2.51 99.44
101829753<br>
512
100000 2.51 111.85 203659506<br>
1024
100000 3.07 106.77 333224861<br>
2048
100000 3.91 105.86 523651240<br>
4096
100000 4.75 101.98 862497367<br>
8192
100000 6.15 100.41 1332899446<br>
16384
100000 8.66 102.84 1891916859<br>
32768
64000 13.97 101.37 2345933562<br>
65536
32000 24.86 102.12 2635884647</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:red'>And after moving to inline send ...</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Using 8 processors. Sender Frequency is 3579545<br>
1
100000 </span><span style='font-size:10.0pt;
font-family:"Arial","sans-serif";color:red'>1.12</span><span style='font-size:
10.0pt;font-family:"Arial","sans-serif";color:blue'>
111.91 895255<br>
2
100000 1.12 118.89
1790510<br>
4 100000
1.12 118.89 3581020<br>
8
100000 1.12 111.91
7162041<br>
16
100000 1.12 118.89
14324082<br>
32
100000 1.12 125.87
28648164<br>
64
100000 1.40 106.30
45845272<br>
128 100000
1.40 111.89 91690544<br>
256
100000 2.23 101.39 114592658</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>