<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
text-align:right;
direction:rtl;
unicode-bidi:embed;
font-size:12.0pt;
font-family:"Times New Roman";}
h1
{margin-top:12.0pt;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:.3in;
text-indent:-.3in;
page-break-after:avoid;
mso-list:l1 level1 lfo1;
font-size:16.0pt;
font-family:Arial;}
h2
{margin-top:12.0pt;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:.4in;
text-indent:-.4in;
page-break-after:avoid;
mso-list:l1 level2 lfo2;
font-size:14.0pt;
font-family:Arial;
font-style:italic;}
h3
{margin-top:12.0pt;
margin-right:.5in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-.5in;
page-break-after:avoid;
mso-list:l1 level3 lfo3;
direction:rtl;
unicode-bidi:embed;
font-size:13.0pt;
font-family:Arial;}
h4
{margin-top:12.0pt;
margin-right:.6in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-.6in;
page-break-after:avoid;
mso-list:l1 level4 lfo4;
direction:rtl;
unicode-bidi:embed;
font-size:14.0pt;
font-family:"Times New Roman";}
h5
{margin-top:12.0pt;
margin-right:.7in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-.7in;
mso-list:l1 level5 lfo5;
direction:rtl;
unicode-bidi:embed;
font-size:13.0pt;
font-family:"Times New Roman";
font-style:italic;}
h6
{margin-top:12.0pt;
margin-right:.8in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-.8in;
mso-list:l1 level6 lfo6;
direction:rtl;
unicode-bidi:embed;
font-size:11.0pt;
font-family:"Times New Roman";}
p.MsoHeading7, li.MsoHeading7, div.MsoHeading7
{margin-top:12.0pt;
margin-right:.9in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-.9in;
mso-list:l1 level7 lfo7;
direction:rtl;
unicode-bidi:embed;
font-size:12.0pt;
font-family:"Times New Roman";}
p.MsoHeading8, li.MsoHeading8, div.MsoHeading8
{margin-top:12.0pt;
margin-right:1.0in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-1.0in;
mso-list:l1 level8 lfo8;
direction:rtl;
unicode-bidi:embed;
font-size:12.0pt;
font-family:"Times New Roman";
font-style:italic;}
p.MsoHeading9, li.MsoHeading9, div.MsoHeading9
{margin-top:12.0pt;
margin-right:1.1in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:right;
text-indent:-1.1in;
mso-list:l1 level9 lfo9;
direction:rtl;
unicode-bidi:embed;
font-size:11.0pt;
font-family:Arial;}
span.MsoEndnoteReference
{vertical-align:super;}
p.MsoEndnoteText, li.MsoEndnoteText, div.MsoEndnoteText
{margin:0in;
margin-bottom:.0001pt;
text-align:right;
direction:rtl;
unicode-bidi:embed;
font-size:10.0pt;
font-family:"Times New Roman";}
p.MsoListBullet, li.MsoListBullet, div.MsoListBullet
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
text-indent:-.25in;
mso-list:l2 level1 lfo11;
font-size:12.0pt;
font-family:"Times New Roman";}
p.MsoTitle, li.MsoTitle, div.MsoTitle
{margin-top:12.0pt;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:center;
direction:rtl;
unicode-bidi:embed;
font-size:16.0pt;
font-family:Arial;
font-weight:bold;}
p.MsoBodyText, li.MsoBodyText, div.MsoBodyText
{margin-top:0in;
margin-right:0in;
margin-bottom:6.0pt;
margin-left:0in;
text-align:right;
direction:rtl;
unicode-bidi:embed;
font-size:12.0pt;
font-family:"Times New Roman";}
p.MsoSubtitle, li.MsoSubtitle, div.MsoSubtitle
{margin-top:0in;
margin-right:0in;
margin-bottom:3.0pt;
margin-left:0in;
text-align:center;
direction:rtl;
unicode-bidi:embed;
font-size:12.0pt;
font-family:Arial;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle23
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:-119;
mso-list-type:simple;
mso-list-template-ids:-156985440;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.25in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.25in;
margin-bottom:0in;
margin-left:.25in;
margin-bottom:.0001pt;
text-indent:-.25in;
font-family:Symbol;}
@list l1
{mso-list-id:1285036741;
mso-list-template-ids:-547819762;}
@list l1:level1
{mso-level-style-link:"Heading 1";
mso-level-text:%1;
mso-level-tab-stop:.3in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.3in;
margin-bottom:0in;
margin-left:.3in;
margin-bottom:.0001pt;
text-indent:-.3in;}
@list l1:level2
{mso-level-style-link:"Heading 2";
mso-level-text:"%1\.%2";
mso-level-tab-stop:.4in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.4in;
margin-bottom:0in;
margin-left:.4in;
margin-bottom:.0001pt;
text-indent:-.4in;}
@list l1:level3
{mso-level-style-link:"Heading 3";
mso-level-text:"%1\.%2\.%3";
mso-level-tab-stop:.5in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.5in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
text-indent:-.5in;}
@list l1:level4
{mso-level-style-link:"Heading 4";
mso-level-text:"%1\.%2\.%3\.%4";
mso-level-tab-stop:.6in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.6in;
margin-bottom:0in;
margin-left:.6in;
margin-bottom:.0001pt;
text-indent:-.6in;}
@list l1:level5
{mso-level-style-link:"Heading 5";
mso-level-text:"%1\.%2\.%3\.%4\.%5";
mso-level-tab-stop:.7in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.7in;
margin-bottom:0in;
margin-left:.7in;
margin-bottom:.0001pt;
text-indent:-.7in;}
@list l1:level6
{mso-level-style-link:"Heading 6";
mso-level-text:"%1\.%2\.%3\.%4\.%5\.%6";
mso-level-tab-stop:.8in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.8in;
margin-bottom:0in;
margin-left:.8in;
margin-bottom:.0001pt;
text-indent:-.8in;}
@list l1:level7
{mso-level-style-link:"Heading 7";
mso-level-text:"%1\.%2\.%3\.%4\.%5\.%6\.%7";
mso-level-tab-stop:.9in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:.9in;
margin-bottom:0in;
margin-left:.9in;
margin-bottom:.0001pt;
text-indent:-.9in;}
@list l1:level8
{mso-level-style-link:"Heading 8";
mso-level-text:"%1\.%2\.%3\.%4\.%5\.%6\.%7\.%8";
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:1.0in;
margin-bottom:0in;
margin-left:1.0in;
margin-bottom:.0001pt;
text-indent:-1.0in;}
@list l1:level9
{mso-level-style-link:"Heading 9";
mso-level-text:"%1\.%2\.%3\.%4\.%5\.%6\.%7\.%8\.%9";
mso-level-tab-stop:1.1in;
mso-level-number-position:left;
mso-margin-top-alt:0in;
margin-right:1.1in;
margin-bottom:0in;
margin-left:1.1in;
margin-bottom:.0001pt;
text-indent:-1.1in;}
@list l2
{mso-list-id:1992904546;
mso-list-type:hybrid;
mso-list-template-ids:1467933110 -81507590 -942524410 67960837 67960833 67960835 67960837 67960833 67960835 67960837;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-style-link:"List Bullet";
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
margin-right:.5in;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
margin-right:1.0in;
text-indent:-.25in;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1 dir=RTL>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Hi
all,<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> I
know I am new to this project and I must be naļve but I want to understand few
things concerning the openib architecture. In the course of learning the openib
gen 2 stack and preparing to port the opensm to it (which is my current task),
I have encountered few areas that seem problematic to me and I would like to
understand the reasoning for it, if not to offer alternatives. I am sorry that
I rise these issues so late, but I was not involved in this project earlier. I
hope it is better late than never.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> It
seems to me that the major design approach is to do everything in the kernel
but let user mode staff access to the lower levels to enable performance sensitive
applications override all kernel layers. Am I right?<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> It
seems also that within the kernel, the ib interface/verbs (ib_*) is very close to
the mthca verbs that are very close to vapi. I know that this is the way most
of the industry was working, but I wonder – is this the correct model? Will
this not pollute the kernel with a lot of IB specific stuff? Personally, I
think that IB verbs (vapi) are so complicated that another level of abstraction
is required. PDs, MRs, QPs QP state machine, PKEYs, MLIDs and other “curses”,
why should a module such as IPoIB knows about it? If the answer is performance
then I have to disagree. In the same fashion you can say that in order to
achieve efficient disk IO applications should know the disks geometry and to
able to do direct IO to the disk firmware, or that applications should talk
SCSI verbs to optimize their data transfers.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> It
seems to me that the current interfaces evolved to what they are today mainly
because of the way IB itself evolved – with a lot of uncertainly and a
lot of design holes (not to say “craters”). This forced most of the
industry to stick with very straight forward interfaces that were based on
Mellanox VAPI.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> I
wonder if this is not the right time to come up with much better abstraction –
for user mode and for kernel mode. For example, it seems that the abstraction layer
should abstract the IB networking objects and not the IB hca interface. In
other words – why not to build the abstraction around IB networking types
– UD, RC, RD, MADS? Why do we have to expose the memory management model
of the driver/HCA to upper layers? Do we really want to expose IB related
structures such as CQs, QPs, and WQE? Why? Not only that this is not good for
abstraction meaning changes in the drivers will require upper layers modifications,
but also this is very problematic due security and stability reasons.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> I
think that using correct abstraction is very critical for a real acceptance in
the Linux/open source world. Good abstraction will enable us also to provide
good and secure kernel mode and user mode interfaces and access.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> Once
we have such interfaces, I think we should consider again the user/kernel
division. As a general rule I think that it is commonly agreed that the kernel
should include only things that must be in the kernel, meaning hardware aware software
and very performance sensitive software. Other software modules may be inserted
to the kernel once it is mature and robust. For example, RPC, NFSD and SMBFS
(SAMBA) were developed in user mode, served many years in user mode and then
after they have matured, they started to “sink” into the kernel. I
think that IB, and especially IB management modules, are far from being mature.
Even the IB standard itself is not really stable. Specifically, there is a
requirement (in the SOW) to make the IB management distributed due some scalability
and other (redundancy, etc.) requirements. I do not know if this requirement
will actually realize, but if is will, the SM and maybe also the SMI/GSI agents
and the CM will have to significantly change. If this is likely to happen, I
would suggest keeping as much as possible in user mode – it is much
easier to develop and to update. We should have kernel based agents and
mechanism to assist the performance, but I think that most of the work should
be done in user mode where it can harm less. Specifically, thinks such as MAD
transaction manager (retries, timeouts, matching), RMPP and other should be developed
in user mode and packed as libraries, again, at least until they will stabilize
and mature. Why should we develop complicated functionality such as RMPP in the
kernel when the only few kernel based queries (if any at all) will use them?<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> If
I not mistaken, one of the IB design goals was to enable efficient *<b><span
style='font-weight:bold'>user mode</span></b>* networking (copy less, low
latency). This is also the major advantage IB have over several alternatives –
most remarkably 10G Ethernet. If we will not emphasize such advantages, we will
reduce the survival chance of IB once 10GE will be widely used. If potential
users will get the impression that comparing to 10GE IB is cheaper, faster,
more efficient but requires tons of special kernel based modules and very
complicated interfaces and therefore much less stable and much more exposed to
bugs, they will use 10GE. I have no doubt. Yes, it is true that this project is
meant to supply HPC code base, but eventually, IB will not survive as HPC
interconnect only. Furthermore, all HPC applications are user mode based. Good
user mode interfaces are critical for HCP not less then to any other high end
networking applications.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> I
really would like to know if I am shooting in the dark or that the issues I
mentioned were discussed and there are good reasons to do them the way they
are. Or, maybe I don’t get the picture and the state of things is
completely different from what I am painting. Either way I would like to know
what do you think.<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal dir=LTR style='text-align:left;direction:ltr;unicode-bidi:
embed'><font size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> Shahar<o:p></o:p></span></font></p>
</div>
</body>
</html>