<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hello list users,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, after getting our use case working from Windows to Windows, I dove into attempting Linux/Windows interoperability with Linux being the sender and Windows being the receiver. The two endpoints connected after accounting for the fact
that the verbs library silently adds and removes a byte at the start of the CM data, while the netdir provider does not.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, the fi_sendv from Linux failed on the netdir side with an ND_DATA_OVERRUN error status. After digging into the netdir provider code a bit, I think I see the source of the problem. It appears that the receive buffer provided in
the call to fi_recvv is not immediately sent to the ND2 implementation, but is queued inside the netdir provider. It appears that the netdir provider expects a small incoming message indicating that a large message transfer is being requested. When processing
this small initiation message, the receive buffer is then processed and passed to the ND2 implementation. Then the large message is transferred and processed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, it appears that some effort will be needed to make changes to the netdir provider in order to support our desired use case. My manager is on board with our making the effort to achieve our desired use case and contributing the code
back upstream. Our concern is that since none of us are particularly experienced with libfabric or Network Direct, we don’t know whether there might be some hard limitation in the Network Direct API that would stop us dead in our tracks from achieving interoperability
between Linux verbs and Windows netdir.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’re wondering whether anyone here knows why the netdir provider might have been designed the way that it is. Or even better, if the original implementers might be available for consultation about their design choices and knowledge about
what limitations we would face in attempting to make changes to achieve our use case.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Many Thanks,<o:p></o:p></p>
<p class="MsoNormal">Derek<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Derek Shinaberry<o:p></o:p></p>
<p class="MsoNormal">Senior Software Engineer, Platform Software<o:p></o:p></p>
<p class="MsoNormal">Canon Medical Research USA, Inc. <br>
706 N. Deerpath Drive, Vernon Hills, IL 60061, USA<br>
<a href="http://www.research.us.medical.canon/"><span lang="EN-AU" style="color:#0563C1">www.</span><span style="color:#0563C1">research.us.medical.canon</span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>