<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>There is a problem with kernel
DLLs in that the dependencies when you update devices is a nightmare to manage,
since the driver doesn’t show up in any device stack. The move away
from a kernel DLL for complib and IBAL was exactly to remove the problems with
doing driver updates, where there wasn’t a good way to make sure that the
IBAL kernel DLL was unloaded before it needed to be updated. For complib,
it was also to allow a mix of release and debug drivers to coexist.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Think of the case where you have
a kernel driver that’s a service, something like SDP. If IBAL was a
DLL, to update the drivers you have to know which services depend on it,
disable all of them, disable all the HCAs, and only then can you actually
update the driver. It’s a system management nightmare.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>In any case, even splitting the
driver up doesn’t solve the problem for user-mode clients like WSD that
currently expect that the IBAL device is available regardless of whether actual
HW is present.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>For kernel clients, the existing
mechanism of querying for the IBAL interface via IRP_MN_ QUERY_INTERFACE
requests should not be changed. This is the proper way for kernel drivers
to get one another’s interfaces, and there is support for PnP
notifications so that drivers can be unloaded and updated. Even querying
the CI interface is fine.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Really, you want to stay away
from kernel DLLs if you can.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>So leave IBAL and the bus driver
together, but make it a filter driver for the HCA. For user-mode devices,
maybe we can also make the ibbus driver a kernel service (managed through the
service manager) so that it gets loaded and creates the well known IBAL device
object for user-mode clients, without requiring a root enumerated device.
I don’t think this solves the issue of upgrading drivers – an admin
will still need to stop the service to update the drivers. This might
also require more code to handle service manager requests. It’s
also possible to make changes in WSD so that the service isn’t needed at
all, so that only applications that depend on it would require the service, and
it would be disabled by default.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>-Fab<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Yossi Leybovich
[mailto:sleybo@mellanox.co.il] <br>
<b>Sent:</b> Thursday, May 10, 2007 5:33 AM<br>
<b>To:</b> Fab Tillier; ofw@lists.openfabrics.org<br>
<b>Subject:</b> RE: [ofw] Loading drivers on LHS<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>I think solution 2 is the way we need to go.</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>For the long term I would like to see IBBUS.sys functionality
split into 2 modules (when we had ibbus.sys and ibal.sys)</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>1. The creation and management of the virtual devices should be
implemented as filter driver over the MTHCA ibbus.sys</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>2. all the VERBS functionality should implemented
as kernel module without any dependence to the HW.</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Thanks</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Yossi </span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<div class=MsoNormal align=center style='text-align:center'><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <b>On Behalf Of </b>Fab Tillier<br>
<b>Sent:</b> Thursday, May 10, 2007 3:19 AM<br>
<b>To:</b> ofw@lists.openfabrics.org<br>
<b>Subject:</b> [ofw] Loading drivers on LHS</span><span style='font-size:12.0pt;
font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal>Hi folks,<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I tried loading the IB drivers on LHS (Windows Server code
name “Longhorn”) and it doesn’t work. I tracked the
problem down to the fact that the driver packages are now staged to a special
system folder before being installed. The IB co-installer uses the path
of the binaries to find the drivers for the “InfiniBand Fabric”
device, but since these didn’t get staged with the HCA drivers, it fails
and with it installation of the HCA drivers fails.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I can’t come up with a really good solution to this,
so wanted to start a discussion for how this should be addressed.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Solution 1: Don’t fail the HCA driver installation if
the co-installer can’t install the IB Bus device.<o:p></o:p></p>
<p class=MsoNormal>While this allows the HCA drivers to install, it
doesn’t install the IB Bus, which means no user-mode app will work (no
IBAL device for it to open), and no IPoIB or other child devices (no bus
driver) will be reported to the system. A manual installation of the
drivers for the IB bus driver is required to get things really running, so
there’s no longer an automatic installation path through the device
manager. Things can be scripted with devcon to work around this, though.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Solution 2: Make ibbus an upper filter for the HCA, include
all its information in the HCA’s INF file.<o:p></o:p></p>
<p class=MsoNormal>This solves the installation issue (and eliminates the
co-installer), and kernel clients will get created as needed, so it’s a
pretty nice solution (it actually simplifies the PnP notifications required of
HCA drivers). The problem is that it impacts user-mode apps.
Currently, a user-mode client can load before any HCA is available in the
system, and use the IBAL PnP notifications to detect arrival of an HCA.
If the IB bus driver is loaded as an upper filter, it will not be loaded until
an HCA is loaded, so user-mode applications will no longer be able to rely on
IBAL for CA PnP notifications. This would impact WSD for sure, though WSD
depends on the IBAT device anyway for address resolution, and this won’t
exist until the IB bus exists. So WSD could be re-architected to
delay-load IBAL until there are addresses reported by IBAT.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Neither solution is all that great. Any other thoughts
and ideas?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>-Fab<o:p></o:p></p>
</blockquote>
</div>
</body>
</html>