Leonid,<br><br>I understand you might be busy with some other stuff too.<br><br>As I already said, this was just a first draft patch and there are no intentions to move it into the svn repository.<br>I am sorry, I guess I misled you by asking the question "whether above patch is acceptable".<br>
<br>This patch was just to show a use case of having multiple child devices associated with a IOC instead of having a single<br>child PDO for each IOC.<br><br>Plan is to provide a IOCTL interface in ibiou for creation of child devices and all device information will be passed through device<br>
IOCTLs (and not INF file, in this first draf patch I used INF file for just POC). So no one has to know the IOC parameters at the time of installation.<br><br>It indeed raise the need of a user application which can scan the fabric and provide list the of reachable IOCs (particularly Port to IOC path, like <b>ibsrpdm</b> and <b>ib_qlgc_vnic_query</b> tools on linux do). <br>
I am thinking of providing a IOCTL in ibiou to return the list of reachable IOCs.<br>Having a list of reachable IOCs with the user, now user can write a configuration file describing the child devices (or a GUI to enter the child device descriptions). This user application will parse the configuration file and will create the device interfaces.<br>
Also it will record the all these descriptions into registry so that from the next boot, ibiou will create the devices without the intervention of user app.<br><br><span>>> <font face="Arial" color="#0000ff" size="2">- having configured one node, one has to repeate all this
work with all the others or to prepare and clone the
image;<br><br><span style="color: rgb(0, 0, 0);">Once a node has been configured with a config file, that can be used on other nodes too.</span><br></font></span><br><span>>> <font face="Arial" color="#0000ff" size="2">- if a new IOC has been added, one has to go through all
the nodes and to change the Registries. It may also require reimaging of all the
nodes;</font></span><br><br>If a new IOC has been added, then yes he would need to make entries into config file and then user app will take care of rest.<br><br><span>>> <font face="Arial" color="#0000ff" size="2">- what will happen during the WINOF package upgrade ? We
don't want user to repeat all this configuration work, do we
?</font></span><br><br>We will have to provide a versioning on above mentioned device IOCTLs interface of ibiou. Minimal release changes will not affect the config file structure and hence from a user perspective reconfiguration will not be required.<br>
<br>This User Application will help create multiple child devices for ULP's (SRP/VNIC) and hence much more failover configurations for the user.<br><br>My basic question is if we agree to above then I can move ahead for implmenting above mentioned device IOCTLs in ibiou<br>
and also start doing work on the developing the user application to do this.<br><br>Please let me know if you have any further queries on it.<br><br>Regards<br>Deepak<br><br><br><div class="gmail_quote">On Thu, Jan 29, 2009 at 3:27 PM, Leonid Keller <span dir="ltr"><<a href="mailto:leonid@mellanox.co.il">leonid@mellanox.co.il</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">
<div><span><font face="Arial" color="#0000ff" size="2">Hi
Deepak,</font></span></div>
<div><span><font face="Arial" color="#0000ff" size="2"></font></span> </div>
<div><span><font face="Arial" color="#0000ff" size="2">Sorry
for that late reaction. Could not find time beforehand to look into it...
:(</font></span></div>
<div><span><font face="Arial" color="#0000ff" size="2"></font></span> </div>
<div><span><font face="Arial" color="#0000ff" size="2">I'm
not sure, whether the advantages of your patch are worth the deployment
problems that it can cause (IMO).</font></span></div>
<div><span><font face="Arial" color="#0000ff" size="2">Here
are several ones at the first look:</font></span></div>
<div><span></span><span> <font face="Arial"><font color="#0000ff" size="2">- one has to know all the IOC parameters at the time of
installation;</font></font></span></div></div></blockquote><div> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div dir="ltr"><div>
<span><font face="Arial"><font color="#0000ff" size="2"></font></font></span></div>
<div><span>
<font face="Arial" color="#0000ff" size="2">And what if they are yet unknown (IOCs
still not purchased) or unknown to the guy (he doesn't know what IOC at all is)
? :(</font></span></div>
<div><span> <font face="Arial"><font color="#0000ff" size="2">- one has to enter IOC parameters during the installation
in some way;</font></font></span></div>
<div><span>
<font face="Arial" color="#0000ff" size="2">Changing INF file is not a convenient way.
One has to provide some GUI to edit the parameters during- and the Registry
after the installation;</font></span></div>
<div><span> <font face="Arial" color="#0000ff" size="2">- having configured one node, one has to repeate all this
work with all the others or to prepare and clone the
image;</font></span></div>
<div><span> <font face="Arial" color="#0000ff" size="2">- if a new IOC has been added, one has to go through all
the nodes and to change the Registries. It may also require reimaging of all the
nodes;</font></span></div>
<div><span> <font face="Arial" color="#0000ff" size="2">- what will happen during the WINOF package upgrade ? We
don't want user to repeat all this configuration work, do we
?</font></span></div>
<div><font face="Arial" color="#0000ff" size="2"></font> </div>
<div><span><font face="Arial" color="#0000ff" size="2">What
do you think ?</font></span></div>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" lang="en-us" align="left">
<hr>
<font face="Tahoma" size="2"><b>From:</b> <a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>
[mailto:<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>] <b>On Behalf Of </b>Deepak
Gupta<br><b>Sent:</b> Tuesday, January 27, 2009 5:19 PM<br><b>To:</b>
<a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> ***SPAM*** Re: [ofw] Expose a
vendor defined device in ibbus?<br></font><br></div><div><div></div><div class="Wj3C7c">
<div></div><font face="Arial" color="#0000ff" size="2"></font><font face="Arial" color="#0000ff" size="2"></font><br>Since I didn't receive any concrete response
from the list about the patch, I still have following questions
unanswered:--<br><br>1) I would like to know who is current owner of ibiou and
get a feedback on whether above patch is acceptable.<br>Or Do you suggest some
modifications on it ?<br><br>2) If we are ok with this patch then what time
frame should we set for it. Should we target it for coming 2.1 release or for
post 2.1 releases?<br><br>3) Also, I want to move ahead for implementing IOCTL
layer in ibiou for creation of child devices and IOC listings on user request.
Is it acceptable to implement this IOCTL layer for device creations and IOC
listings in ibiou and you suggest to achieve the same through some other
means?<br><br><br><b>I want to reiterate the purpose of this
patch:--</b><br><br>This patch is for creating multiple PDO's for a HCA to IOC
path as configured by the user in INF file.<br>It will help in having multiple
sessions with the IOC and hence will give some user configurations for fail
over.<br><br><b>Behavior change from current ibiou
implementation:--</b><br><br>Currently ibiou simply discovers the IOCs and
creates child devices for the IOCs discovered.<br>This patch will create
multiple child devices for HCA to IOC paths as configured by the user in INF
file.<br>If there are no child device configurations for a particular IOC then
no devices will be created for that IOC and hence will not appear in device
manager's device tree.<br><br>This is just a first draft patch to show the
usability and we have plans to implement IOCTLs in ibiou for child device
creations and for listing the reachable IOC PATHs so that a user application
can be written which can list the IOCs and local HCAs to the user and user can
send IOCTLs to create devices for the
ULPs.<br><br><br>Regards<br>Deepak<br><br><br><br><br>
<div class="gmail_quote">On Tue, Jan 27, 2009 at 7:09 PM, Deepak Gupta <span dir="ltr"><<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Forgot
to include the list.<br><br>Regards<br><font color="#888888">Deepak<br><br></font>
<div class="gmail_quote">
<div>---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Deepak Gupta</b> <span dir="ltr"><<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>></span><br>
Date: Sat, Jan 24,
2009 at 5:07 PM<br>Subject: Re: [ofw] Expose a vendor defined device in
ibbus?<br></div>
<div>
<div></div>
<div>To: Fab Tillier <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>><br><br><br>Fab, Nice to
see your response.<br>Please see below.<br><br>
<div class="gmail_quote">
<div>On Fri, Jan 23, 2009 at 11:09 PM, Fab Tillier <span dir="ltr"><<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div vlink="purple" link="blue" lang="EN-US">
<div>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">I don't have time
to look at this in depth, but I can tell you from past experience that
having a single PDO for an IOC leads to trouble if you ever have multiple
HCAs in the system. If you have a miniport driver (like NDIS or
StorPort), the port driver takes care of DMA mappings of user-provided
buffers. That DMA mapping goes down to the PCI driver for the
particular HCA, so you could potentially have a mapping that isn't valid
for one of the multiple HCAs in the system. It's best IMO to have a
PDO per IOC per HCA (this allows automatic path migration to work in
multi-port HCAs), and push failover between IOCs to a higher level
(LBFO/MPIO).</span></p></div></div></blockquote></div>
<div><br>I had the same concern while implementing the patch.<br>So I tried
to dig into the stack till h/w drivers and it came to me that all DMA
mappings are being<br>handled by PCI driver.<br>So this patch creates PDO's
per IOC per HCA. <br>User can specify the child device descriptions in INF
file (later on we can think of having a IOCTL interface in ibiou) and HCA to
IOC path.<br>ibiou will create child PDO's only if HCA to IOC path is
present.<br><br>Any more suggestions are welcome!!!<br><br>Regards<br><font color="#888888">Deepak<br> <br></font></div>
<div>
<div></div>
<div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div vlink="purple" link="blue" lang="EN-US">
<div>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span></p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span> </p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">-Fab</span></p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span> </p>
<div style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1.5pt; padding: 0in 0in 0in 4pt;">
<div>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0in 0in;">
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> <a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a> [mailto:<a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a>] <b>On Behalf Of </b>Deepak
Gupta<br><b>Sent:</b> Friday, January 23, 2009 6:41 AM<br><b>To:</b> James
Yang<br><b>Cc:</b> Leonid Keller; Fab Tillier; <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a>
<div>
<div></div>
<div><br><b>Subject:</b> Re: [ofw] Expose a vendor defined device in
ibbus?</div></div></span>
</p><p></p></div></div>
<div>
<div></div>
<div>
<p> </p>
<p style="margin-bottom: 12pt;">Did any one get the chance to have a look
at the patch?<br><br>I am really concerned with the tight coupling of
representing each single IOC as a single
PDO.<br><br>Regards<br>Deepak<br><br><br></p>
<div>
<p>On Wed, Jan 21, 2009 at 5:55 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p style="margin-bottom: 12pt;">All,<br><br>Based on James patch of
creating user defined devices tied to local HCA ports, I have created a
first draft patch attached herewith <br>to allow "ibiou" to create user
defined devices which will associate with IOC paths.<br><br>This patch
does the following:-<br><br>Now ibiou will not create child devices on
it's own when it discovers a IOC.<br>Instead it will create child devices
only when it is instructed to do so.<br>For the sake of this patch,
currently it uses name/description of the devices hard coded into INF file
of ibiou driver.<br>Later on we can implement a IOCTL to pass this
information to the driver (more below)<br><br>It will make a listing of
devices to be created in it's DriverEntry routine.<br>When it will get
IOC_PNP_ADD pnp events, it will check in device list prepare<br>earlier.
If it finds device(s) in it's list which corresponds to the new
IOC<br>discovered then it will create those child devices and a similar
mechanism<br>will happen when ibiou will get IOC_PNP_ADD
event.<br><br>This more of a initial patch for giving an insight into what
we want to achieve.<br>On a longer run we are thinking of providing a
IOCTL support in ibiou driver for creating child devices (like
vnic/srp/etc) and<br>differnet vendor specific applications can create
those devices and have their driver loaded on top of
that.<br><br>Following is the advantage of this
functionality:-<br><br> Earlier there was only one device
created per IOC on host side.<br> Now there can be multiple
devices on host side per IOC as configured in the
registry.<br> It will be many to one function i.e there can be
many devices on host side which will target to the same
IOC.<br> It will be like having different sessions with the
same IOC.<br><br>Please have a look at it and let me know of your
comments/suggestions/feeback.<br><br>NOTE--> Device descriptions in the
INF file of this patch creates devices that corresponds to a IOC PATH
(CAGUID and IOCGUID).<br> Those
values are local to my machine and you should change it according to your
fabric.<br><br>Regards<br><span style="color: rgb(136, 136, 136);">Deepak<br><br><br></span></p>
<div>
<div>
<div>
<p>On Tue, Jan 20, 2009 at 1:37 AM, James Yang <<a href="mailto:jyang@xsigo.com" target="_blank">jyang@xsigo.com</a>>
wrote:</p></div></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1pt; padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;">
<div>
<div>
<div>
<div>
<p><span style="font-size: 10pt; color: navy;">The proposal is to use
registry key to define vendor devices, and the registry key is global to
the driver. The assumption is that multiple HCA cards will have the same
vendor defined devices. It cannot support one HCA with vendor-A device,
and the other HCA with vendor-B device, in the same system. At default
all hcas will have IPoIB as child devices.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Thanks,</span></p>
<p><span style="font-size: 10pt; color: navy;">James</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<div>
<div style="text-align: center;" align="center">
<hr align="center" size="2" width="100%">
</div>
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Leonid Keller [mailto:<a href="mailto:leonid@mellanox.co.il" target="_blank">leonid@mellanox.co.il</a>] <br><b>Sent:</b> Sunday,
January 18, 2009 5:57 AM<br><b>To:</b> James Yang; Deepak Gupta; Fab
Tillier</span></p>
<div>
<div>
<p><span style="font-size: 10pt;"><br><b>Cc:</b> <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> RE: [ofw]
Expose a vendor defined device in ibbus?</span></p></div></div></div>
<div>
<div>
<p> </p>
<div>
<p><span style="font-size: 10pt; color: blue;">After first look: why did
you put the list of the created devices into Globals (and not,
say, FDO) ?</span></p></div>
<div>
<p><span style="font-size: 10pt; color: blue;">How it will work for
multi-home machine ? (several HCA cards)</span></p></div>
<div>
<p><span style="font-size: 10pt; color: blue;">Would anybode like to
check it for various partition keys ?</span></p></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1.5pt; margin: 5pt 0in 5pt 3.75pt; padding: 0in 0in 0in 4pt;">
<p> </p>
<div style="text-align: center;" align="center">
<hr align="center" size="2" width="100%">
</div>
<p style="margin-bottom: 12pt;"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;">
<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>] <b>On Behalf Of
</b>James Yang<br><b>Sent:</b> Tuesday, January 06, 2009 10:11
PM<br><b>To:</b> Deepak Gupta; Fab Tillier<br><b>Cc:</b> <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> RE:
[ofw] Expose a vendor defined device in ibbus?</span></p>
<p><span style="font-size: 10pt; color: navy;">Hi,</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Please review the patch
to create user defined devices by reading from registry. By default
there is only one IpoIB device enabled in mlx4_hca.inx file. This
patch will only work for ConnectX.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">The paritition key if
set to default to FFFF, I didn't test on other value. And the Ioctl
part to add partition key may also need to be verified.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Thanks,</span></p>
<p><span style="font-size: 10pt; color: navy;">James</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<div>
<div style="text-align: center;" align="center">
<hr align="center" size="2" width="100%">
</div>
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> <a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a> [mailto:<a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a>] <b>On Behalf Of </b>Deepak
Gupta<br><b>Sent:</b> Monday, January 05, 2009 10:51 PM<br><b>To:</b>
Fab Tillier<br><b>Cc:</b> Tzachi Dar; James Yang; <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> Re:
[ofw] Expose a vendor defined device in ibbus?</span></p></div>
<p> </p>
<p style="margin-bottom: 12pt;">Have a gr8 New Year to all the
members!!!.<br><br>Do you we any updates on "vendor defined device in
ibbus"?<br><br>I wanted to create multiple vnic interfaces
irrespective of number of reachable IOCs.<br><br>Currently I am
creating vnic child devices on "root" bus.<br>Inside VNIC driver, I am
looking for arrival GUID_IB_AL_INTERFACE and hence contacting the
IBAL.<br>But since vnic devices are root enumerated, VNIC driver gets
loaded very earlier in boot phase (Before "Extended Base" group to
which IB Stack drivers belong) and hence VNIC device interfaces are
not getting initialized properly.<br><br>If we are providing a vendor
defined device functionality in ibbus in near future, then it would be
worth for me to wait for it.<br><br>Can any one please comment on
this.<br><br>Regards<br>Deepak</p>
<div>
<p>On Mon, Dec 22, 2008 at 1:12 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p>All,<br><br>I came across one more question in my mind which are I
think is not clear to me after reading the whole thread.<br><br>In new
design being discussed:- <br>Are we making sure that we can have more
than one child devices configured for the same IOC.<br>Currently,
there is one child device created per IOC discovered.<br><br>Having
more than one child device configured for same IOC is a requirement if
a user wants two different ULP interfaces to be created on host
side.<br>Consider a case in which a host is connected to a single IOC
and IOC is connected to a ethernet network via switch.<br>If there are
two different IP subnets then there is a requirement of two different
Ethernet interfaces on the host side too.<br><br>Please let me know if
you need more clarification of my question.<br><br>Regards<br><span style="color: rgb(136, 136, 136);">Deepak</span></p>
<div>
<div>
<p style="margin-bottom: 12pt;"> </p>
<div>
<p>On Sat, Dec 20, 2008 at 2:19 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p style="margin-bottom: 12pt;">Please see
below.<br><br>Regards<br>Deepak</p>
<div>
<div>
<div>
<p>On Sat, Dec 20, 2008 at 12:42 AM, Fab Tillier <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>> wrote:</p>
<div>
<p>> On Wed, Dec 17, 2008 at 11:43 PM, Fab Tillier<br>> <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>> wrote:</p></div>
<div>
<p style="margin-bottom: 12pt;">>> Are there other properties
that I have missed that are needed?<br>><br>> We need a way in
which devices created should be configured for<br>> failovers (
ULPs like VNIC, SRP need more configurable failovers).<br>> Looking
at IBAL's code it create the devices based on the reachable<br>>
IOC's and thus failover's are possible across the HCA/ports and
not<br>> across two different IOCs.<br>> Users can have a case
in which two different IOCs connected to same<br>> physical
network/storage (redundancy is provided for high availability)<br>>
and want a failover across the IOCs.</p></div>
<p>This would be done via LBFO for network devices, and MPIO for
storage devices. I think having the bus driver report a single
IOC that really maps to two IOCs on the fabric is asking for
management problems. Further, LBFO/MPIO can provide failover
between different device types, so the failover devices don't have to
be identical HW.</p></div></div>
<div>
<p><br>I don't know about how MPIO works. But for LBFO, BundleID
param will have to be included in extended params then so that user
gets the freedom of bundling different failover
configurations.<br> </p></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1pt; margin: 5pt 0in 5pt 4.8pt; padding: 0in 0in 0in 6pt;">
<p><br><span style="color: rgb(136, 136, 136);"><br>-Fab</span></p></blockquote></div>
<p> </p></div>
<p> </p></div></div></div>
<p> </p></blockquote></div></div></div></div>
<p> </p></div></div>
<div>
<p>_______________________________________________<br>ofw mailing
list<br><a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a></p>
</div></blockquote></div>
<p> </p></div>
<p> </p></div></div></div></div></div><br>_______________________________________________<br>ofw
mailing list<br><a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a><br>
</blockquote></div></div></div><br></div></div></div><br></blockquote></div><br></div></div></blockquote></div>
<br>_______________________________________________<br>
ofw mailing list<br>
<a href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a><br></blockquote></div><br>