<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Christoph. I know just what you mean about fragmented ideas, so I encourage the email discussion as a means of bringing some cohesion. Then we can
use next Tuesday’s meeting to bring the group up to speed and help add some shape to it to see if we can force a proposal to emerge.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> christoph@graphe.net [mailto:christoph@graphe.net]
<b>On Behalf Of </b>Christoph Lameter<br>
<b>Sent:</b> Thursday, December 05, 2013 12:40 PM<br>
<b>To:</b> Paul Grun<br>
<b>Cc:</b> 'openframeworkwg@lists.openfabrics.org' (openframeworkwg@lists.openfabrics.org)<br>
<b>Subject:</b> Re: [Openframeworkwg] Some concerns about the new Fabric/IBverbs API<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Well my ideas are a bit fragmented right now and I am not sure that I can even call this a proposal (more vague ideas) but I can try to get some more cohesive thoughts together by Tuesday. I have to confess
that I am a bit more comfortable with developing ideas via email since that is the traditional mode of operation among kernel developers.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I think we need to explore way to simplify the use of RDMA and at the same time make it more universally applicable. Using established mechanisms like the library system and existing raw device interfaces could
be a viable approach.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I just looked at the inline code in ibverbs and it looks like this is simply a move of the indirect function call into the app instead of being in separate function. That is not really what I would call inline.
A true inline call would be able to hand the device i/o without any additional function calls. With libraries true inline calls would be possible if one would compile against a specific vendor library. A vendor could provide an inline function f.e. to poll
the cq by reading/writing memory mapped location directly from within the app.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">If one has done so then the application would be only able to work with a certain device since device specific inline code was generated in the executable.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">If we just want the indirection inline then that can also be done with a library approach by setting up vectors there. dlopen and friends can then be used to populate that table and the same approach that ibverbs
does today would still work.<o:p></o:p></p>
</div>
<p class="MsoNormal">How much of a benefit is the current inline indirection implementation in ibverbs.h for ib_post_send etc?
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Dec 5, 2013 at 1:15 PM, Paul Grun <<a href="mailto:grun@cray.com" target="_blank">grun@cray.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am replying to Christoph’s initial message because it seems to frame the question most clearly.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Christoph, could I persuade you to present your thoughts in the form of a brief proposal? That might
be a good way to frame the question in order to bring the whole up to speed on your idea as well as accelerate the process of coming to closure. It need not be a detailed proposal, but simply a way to both fully understand the idea and understand its merits.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Correct me if I am wrong, but I think you are proposing two separate, but interrelated ideas:
</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Leverage existing Linux shared library mechanisms as a “pseudo-framework” for providing application access to provider services
</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The use of the VFIO interface to all direct memory access transfers (including direct access to low level driver controls).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Is this something you would be willing to take on for next Tuesday’s meeting?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Paul</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:openframeworkwg-bounces@lists.openfabrics.org" target="_blank">openframeworkwg-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:openframeworkwg-bounces@lists.openfabrics.org" target="_blank">openframeworkwg-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Christoph Lameter<br>
<b>Sent:</b> Wednesday, December 04, 2013 8:59 AM<br>
<b>To:</b> '<a href="mailto:openframeworkwg@lists.openfabrics.org" target="_blank">openframeworkwg@lists.openfabrics.org</a>' (<a href="mailto:openframeworkwg@lists.openfabrics.org" target="_blank">openframeworkwg@lists.openfabrics.org</a>)<br>
<b>Subject:</b> [Openframeworkwg] Some concerns about the new Fabric/IBverbs API</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">One of the things that worries me is that we develop specialized APIs based on function vectors.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">That kind functionality is already provided by the shared library systems in Linux. Could not be each of the modules that we talked about just be mapped to a shared library and define
the API simply that way? There are mechanisms in place to deal with compatibility issues.
<br>
<br>
What are the specific requirements that prevent us from using the standard library approach? If at all possible we should simply use what is there.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Then regarding the separation between user space and kernel space: There are alternate approaches possible to the use of IBverbs. The Linux kernel has added a VFIO interface that allows
exposure of device driver controls to user space. If we could base libfabrics on that API instead then the RDMA approach would be generically be usable with any device that supports VFIO.<br>
<br>
<a href="http://lwn.net/Articles/509153/" target="_blank">http://lwn.net/Articles/509153/</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">We could slim down the components needed in the kernel and provide some simpler management APIs. The IBverbs code per se could be in userspace.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">In fact in our company we have some tentative ideas that we could avoid going through IBverbs by going through VFIO. The problem there is that no standard user space APIs exist to interact
with a device. Having such an API would allow a generic way to use RDMA with devices. We have a couple o vendors that are already providing some simplistic proprietary libraries here. Standardization would be good.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>