Stan,<br><br>Thanks for response.<br><br>Yes you are correct, we will create a named device object with user space visible symbolic link to it.<br>We will send IOCTLs to the this device for child device creations (as configured by user).<br>
As explained in previous mails by Alex and I, we will need this feature QLogic EVIC to give user a configuration option.<br>If there are concerns over user configuration of SRP PDO's, then we will restrict the implementation to QLogic EVIC's IOC's only.<br>
And device creations for other ULP's (SRP) will behave as it is used to behave.<br><br>Regards<br>Deepak<br><br><br><div class="gmail_quote">On Thu, Jan 29, 2009 at 10:45 PM, Smith, Stan <span dir="ltr"><<a href="mailto:stan.smith@intel.com">stan.smith@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div><font face="Arial" size="2" color="#0000ff"><span>Please
see inline comments.</span></font></div><br>
<div dir="ltr" lang="en-us" align="left">
<hr>
<font face="Tahoma" size="2"><b>From:</b> <a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>
[mailto:<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>] <b>On Behalf Of </b>Deepak
Gupta<br><b>Sent:</b> Tuesday, January 27, 2009 7:19 AM<br><b>To:</b>
<a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> ***SPAM*** Re: [ofw] Expose a
vendor defined device in ibbus?<br></font><br></div><div class="Ih2E3d">
<div></div>
<div><br>Since I didn't receive any concrete response from the list about the
patch, I still have following questions unanswered:--<br><br>1) I would like to
know who is current owner of ibiou and get a feedback on whether above patch is
acceptable.<br>Or Do you suggest some modifications on it ?<span><font face="Arial" size="2" color="#0000ff"> </font></span></div>
<div><span></span> </div>
</div><div><span><font face="Arial" size="2" color="#0000ff">See
trunk\docs\maintainers.txt</font> <font face="Arial" size="2" color="#0000ff">or
<a href="http://www.openfabrics.org/downloads/WinOF/README" target="_blank">http://www.openfabrics.org/downloads/WinOF/README</a></font></span><div class="Ih2E3d"><br><br>2)
If we are ok with this patch then what time frame should we set for it. Should
we target it for coming 2.1 release or for post 2.1 releases?<span><font face="Arial" size="2" color="#0000ff"> </font></span></div></div>
<div><span></span> </div>
<div><span><font face="Arial" size="2" color="#0000ff">WinOF
2.1 freezes functionality in April; see <a href="http://www.openfabrics.org/downloads/WinOF/WinOF_Roadmap.txt" target="_blank">http://www.openfabrics.org/downloads/WinOF/WinOF_Roadmap.txt</a></font></span></div></div></blockquote>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div><span><font face="Arial" size="2" color="#0000ff"><a href="http://www.openfabrics.org/downloads/WinOF/WinOF_Roadmap.txt" target="_blank"></a></font></span><div class="Ih2E3d">
<br><br>3)
Also, I want to move ahead for implementing IOCTL layer in ibiou for creation of
child devices and IOC listings on user request. Is it acceptable to implement
this IOCTL layer for device creations and IOC listings in ibiou and you suggest
to achieve the same through some other means?<span><font face="Arial" size="2" color="#0000ff"> </font></span></div></div>
<div><span><font face="Arial" size="2" color="#0000ff"></font></span> </div>
<div><span><font face="Arial" size="2" color="#0000ff">By
IOCTL layer are you implying the creation of a user visible device to which a
user-app will open->ioctl-> close?</font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff">Said
IOC device is implemented in ibiou.sys
driver?</font> </span><div class="Ih2E3d"><br><br><br><b>I want to reiterate the purpose of
this patch:--</b><br><br>This patch is for creating multiple PDO's for a HCA to
IOC path as configured by the user in INF file.<br>It will help in having
multiple sessions with the IOC and hence will give some user configurations for
fail over.<br><br><b>Behavior change from current ibiou
implementation:--</b><br><br>Currently ibiou simply discovers the IOCs and
creates child devices for the IOCs discovered.<br>This patch will create
multiple child devices for HCA to IOC paths as configured by the user in INF
file.<br>If there are no child device configurations for a particular IOC then
no devices will be created for that IOC and hence will not appear in device
manager's device tree.<span><font face="Arial" size="2" color="#0000ff"> </font></span></div></div>
<div><span><font face="Arial" size="2" color="#0000ff"></font></span> </div>
<div><span><font face="Arial" size="2" color="#0000ff">I'm no
IOU expert so please bare with me on this...</font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff"></font></span> </div>
<div><span><font face="Arial" size="2" color="#0000ff">Your
implication here is the assumption of prior knowledge of which IOCs will be
present on the fabric.</font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff">If I have an OFED SRP target then I will need a specific entry
in the ibiou.inf file to describe the SRP target?</font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff">Seems
to be rather cumbersome. Particularly from a package installation perspective.
If a .msi installer chooses the SRP option, then how does said option get into
the ibiou.inf file? Is the SRP entry always present in the .inf
file?</font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff">If the
SRP device does not appear then how will it's device driver be
installed?</font></span></div>
<div><span></span><span><font face="Arial" size="2" color="#0000ff"> </font></span></div>
<div><span><font face="Arial" size="2" color="#0000ff">How
would IPoIB work?</font></span></div>
<div><span></span> </div>
<div><span><font face="Arial" size="2" color="#0000ff">All in
all, if I am understanding your design, it sounds like you are
defeating the fabric discovery functionality which I believe to be a very
desirable feature.</font> </span><div><div></div><div class="Wj3C7c"><br><br>This is just a first draft patch
to show the usability and we have plans to implement IOCTLs in ibiou for child
device creations and for listing the reachable IOC PATHs so that a user
application can be written which can list the IOCs and local HCAs to the user
and user can send IOCTLs to create devices for the
ULPs.<br><br><br>Regards<br>Deepak<br><br><br><br><br></div></div></div><div><div></div><div class="Wj3C7c">
<div class="gmail_quote">On Tue, Jan 27, 2009 at 7:09 PM, Deepak Gupta <span dir="ltr"><<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Forgot
to include the list.<br><br>Regards<br><font color="#888888">Deepak<br><br></font>
<div class="gmail_quote">
<div>---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Deepak Gupta</b> <span dir="ltr"><<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>></span><br>
Date: Sat, Jan 24,
2009 at 5:07 PM<br>Subject: Re: [ofw] Expose a vendor defined device in
ibbus?<br></div>
<div>
<div></div>
<div>To: Fab Tillier <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>><br><br><br>Fab, Nice to
see your response.<br>Please see below.<br><br>
<div class="gmail_quote">
<div>On Fri, Jan 23, 2009 at 11:09 PM, Fab Tillier <span dir="ltr"><<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div vlink="purple" link="blue" lang="EN-US">
<div>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">I don't have time to
look at this in depth, but I can tell you from past experience that having a
single PDO for an IOC leads to trouble if you ever have multiple HCAs in the
system. If you have a miniport driver (like NDIS or StorPort), the
port driver takes care of DMA mappings of user-provided buffers. That
DMA mapping goes down to the PCI driver for the particular HCA, so you could
potentially have a mapping that isn't valid for one of the multiple HCAs in
the system. It's best IMO to have a PDO per IOC per HCA (this allows
automatic path migration to work in multi-port HCAs), and push failover
between IOCs to a higher level
(LBFO/MPIO).</span></p></div></div></blockquote></div>
<div><br>I had the same concern while implementing the patch.<br>So I tried to
dig into the stack till h/w drivers and it came to me that all DMA mappings
are being<br>handled by PCI driver.<br>So this patch creates PDO's per IOC per
HCA. <br>User can specify the child device descriptions in INF file (later on
we can think of having a IOCTL interface in ibiou) and HCA to IOC
path.<br>ibiou will create child PDO's only if HCA to IOC path is
present.<br><br>Any more suggestions are welcome!!!<br><br>Regards<br><font color="#888888">Deepak<br> <br></font></div>
<div>
<div></div>
<div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div vlink="purple" link="blue" lang="EN-US">
<div>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span></p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span> </p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">-Fab</span></p>
<p><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span> </p>
<div style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1.5pt; padding: 0in 0in 0in 4pt;">
<div>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0in 0in;">
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> <a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a> [mailto:<a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a>]
<b>On Behalf Of </b>Deepak Gupta<br><b>Sent:</b> Friday, January 23, 2009
6:41 AM<br><b>To:</b> James Yang<br><b>Cc:</b> Leonid Keller; Fab Tillier;
<a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a>
<div>
<div></div>
<div><br><b>Subject:</b> Re: [ofw] Expose a vendor defined device in
ibbus?</div></div></span>
</p><p></p></div></div>
<div>
<div></div>
<div>
<p> </p>
<p style="margin-bottom: 12pt;">Did any one get the chance to have a look at
the patch?<br><br>I am really concerned with the tight coupling of
representing each single IOC as a single
PDO.<br><br>Regards<br>Deepak<br><br><br></p>
<div>
<p>On Wed, Jan 21, 2009 at 5:55 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p style="margin-bottom: 12pt;">All,<br><br>Based on James patch of creating
user defined devices tied to local HCA ports, I have created a first draft
patch attached herewith <br>to allow "ibiou" to create user defined devices
which will associate with IOC paths.<br><br>This patch does the
following:-<br><br>Now ibiou will not create child devices on it's own when
it discovers a IOC.<br>Instead it will create child devices only when it is
instructed to do so.<br>For the sake of this patch, currently it uses
name/description of the devices hard coded into INF file of ibiou
driver.<br>Later on we can implement a IOCTL to pass this information to the
driver (more below)<br><br>It will make a listing of devices to be created
in it's DriverEntry routine.<br>When it will get IOC_PNP_ADD pnp events, it
will check in device list prepare<br>earlier. If it finds device(s) in it's
list which corresponds to the new IOC<br>discovered then it will create
those child devices and a similar mechanism<br>will happen when ibiou will
get IOC_PNP_ADD event.<br><br>This more of a initial patch for giving an
insight into what we want to achieve.<br>On a longer run we are thinking of
providing a IOCTL support in ibiou driver for creating child devices (like
vnic/srp/etc) and<br>differnet vendor specific applications can create those
devices and have their driver loaded on top of that.<br><br>Following is the
advantage of this functionality:-<br><br> Earlier there was only
one device created per IOC on host side.<br> Now there can be
multiple devices on host side per IOC as configured in the
registry.<br> It will be many to one function i.e there can be
many devices on host side which will target to the same IOC.<br>
It will be like having different sessions with the same IOC.<br><br>Please
have a look at it and let me know of your
comments/suggestions/feeback.<br><br>NOTE--> Device descriptions in the
INF file of this patch creates devices that corresponds to a IOC PATH
(CAGUID and IOCGUID).<br> Those
values are local to my machine and you should change it according to your
fabric.<br><br>Regards<br><span style="color: rgb(136, 136, 136);">Deepak<br><br><br></span></p>
<div>
<div>
<div>
<p>On Tue, Jan 20, 2009 at 1:37 AM, James Yang <<a href="mailto:jyang@xsigo.com" target="_blank">jyang@xsigo.com</a>>
wrote:</p></div></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1pt; padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;">
<div>
<div>
<div>
<div>
<p><span style="font-size: 10pt; color: navy;">The proposal is to use
registry key to define vendor devices, and the registry key is global to
the driver. The assumption is that multiple HCA cards will have the same
vendor defined devices. It cannot support one HCA with vendor-A device,
and the other HCA with vendor-B device, in the same system. At default all
hcas will have IPoIB as child devices.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Thanks,</span></p>
<p><span style="font-size: 10pt; color: navy;">James</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<div>
<div style="text-align: center;" align="center">
<hr size="2" width="100%" align="center">
</div>
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Leonid Keller [mailto:<a href="mailto:leonid@mellanox.co.il" target="_blank">leonid@mellanox.co.il</a>] <br><b>Sent:</b> Sunday, January
18, 2009 5:57 AM<br><b>To:</b> James Yang; Deepak Gupta; Fab
Tillier</span></p>
<div>
<div>
<p><span style="font-size: 10pt;"><br><b>Cc:</b> <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> RE: [ofw]
Expose a vendor defined device in ibbus?</span></p></div></div></div>
<div>
<div>
<p> </p>
<div>
<p><span style="font-size: 10pt; color: blue;">After first look: why did
you put the list of the created devices into Globals (and not,
say, FDO) ?</span></p></div>
<div>
<p><span style="font-size: 10pt; color: blue;">How it will work for
multi-home machine ? (several HCA cards)</span></p></div>
<div>
<p><span style="font-size: 10pt; color: blue;">Would anybode like to check
it for various partition keys ?</span></p></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1.5pt; margin: 5pt 0in 5pt 3.75pt; padding: 0in 0in 0in 4pt;">
<p> </p>
<div style="text-align: center;" align="center">
<hr size="2" width="100%" align="center">
</div>
<p style="margin-bottom: 12pt;"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;">
<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>] <b>On Behalf Of
</b>James Yang<br><b>Sent:</b> Tuesday, January 06, 2009 10:11
PM<br><b>To:</b> Deepak Gupta; Fab Tillier<br><b>Cc:</b> <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> RE: [ofw]
Expose a vendor defined device in ibbus?</span></p>
<p><span style="font-size: 10pt; color: navy;">Hi,</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Please review the patch to
create user defined devices by reading from registry. By default there
is only one IpoIB device enabled in mlx4_hca.inx file. This patch will
only work for ConnectX.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">The paritition key if set
to default to FFFF, I didn't test on other value. And the Ioctl part to
add partition key may also need to be verified.</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;">Thanks,</span></p>
<p><span style="font-size: 10pt; color: navy;">James</span></p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<p><span style="font-size: 10pt; color: navy;"></span> </p>
<div>
<div style="text-align: center;" align="center">
<hr size="2" width="100%" align="center">
</div>
<p><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> <a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a> [mailto:<a href="mailto:mailmeatdkg@gmail.com" target="_blank">mailmeatdkg@gmail.com</a>] <b>On Behalf Of </b>Deepak
Gupta<br><b>Sent:</b> Monday, January 05, 2009 10:51 PM<br><b>To:</b>
Fab Tillier<br><b>Cc:</b> Tzachi Dar; James Yang; <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><b>Subject:</b> Re: [ofw]
Expose a vendor defined device in ibbus?</span></p></div>
<p> </p>
<p style="margin-bottom: 12pt;">Have a gr8 New Year to all the
members!!!.<br><br>Do you we any updates on "vendor defined device in
ibbus"?<br><br>I wanted to create multiple vnic interfaces irrespective
of number of reachable IOCs.<br><br>Currently I am creating vnic child
devices on "root" bus.<br>Inside VNIC driver, I am looking for arrival
GUID_IB_AL_INTERFACE and hence contacting the IBAL.<br>But since vnic
devices are root enumerated, VNIC driver gets loaded very earlier in
boot phase (Before "Extended Base" group to which IB Stack drivers
belong) and hence VNIC device interfaces are not getting initialized
properly.<br><br>If we are providing a vendor defined device
functionality in ibbus in near future, then it would be worth for me to
wait for it.<br><br>Can any one please comment on
this.<br><br>Regards<br>Deepak</p>
<div>
<p>On Mon, Dec 22, 2008 at 1:12 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p>All,<br><br>I came across one more question in my mind which are I
think is not clear to me after reading the whole thread.<br><br>In new
design being discussed:- <br>Are we making sure that we can have more
than one child devices configured for the same IOC.<br>Currently, there
is one child device created per IOC discovered.<br><br>Having more than
one child device configured for same IOC is a requirement if a user
wants two different ULP interfaces to be created on host
side.<br>Consider a case in which a host is connected to a single IOC
and IOC is connected to a ethernet network via switch.<br>If there are
two different IP subnets then there is a requirement of two different
Ethernet interfaces on the host side too.<br><br>Please let me know if
you need more clarification of my question.<br><br>Regards<br><span style="color: rgb(136, 136, 136);">Deepak</span></p>
<div>
<div>
<p style="margin-bottom: 12pt;"> </p>
<div>
<p>On Sat, Dec 20, 2008 at 2:19 PM, Deepak Gupta <<a href="mailto:deepak.gupta@qlogic.com" target="_blank">deepak.gupta@qlogic.com</a>> wrote:</p>
<p style="margin-bottom: 12pt;">Please see
below.<br><br>Regards<br>Deepak</p>
<div>
<div>
<div>
<p>On Sat, Dec 20, 2008 at 12:42 AM, Fab Tillier <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>> wrote:</p>
<div>
<p>> On Wed, Dec 17, 2008 at 11:43 PM, Fab Tillier<br>> <<a href="mailto:ftillier@windows.microsoft.com" target="_blank">ftillier@windows.microsoft.com</a>> wrote:</p></div>
<div>
<p style="margin-bottom: 12pt;">>> Are there other properties that
I have missed that are needed?<br>><br>> We need a way in which
devices created should be configured for<br>> failovers ( ULPs like
VNIC, SRP need more configurable failovers).<br>> Looking at IBAL's
code it create the devices based on the reachable<br>> IOC's and thus
failover's are possible across the HCA/ports and not<br>> across two
different IOCs.<br>> Users can have a case in which two different
IOCs connected to same<br>> physical network/storage (redundancy is
provided for high availability)<br>> and want a failover across the
IOCs.</p></div>
<p>This would be done via LBFO for network devices, and MPIO for storage
devices. I think having the bus driver report a single IOC that
really maps to two IOCs on the fabric is asking for management problems.
Further, LBFO/MPIO can provide failover between different device
types, so the failover devices don't have to be identical
HW.</p></div></div>
<div>
<p><br>I don't know about how MPIO works. But for LBFO, BundleID
param will have to be included in extended params then so that user gets
the freedom of bundling different failover
configurations.<br> </p></div>
<blockquote style="border-style: none none none solid; border-color: -moz-use-text-color; border-width: medium medium medium 1pt; margin: 5pt 0in 5pt 4.8pt; padding: 0in 0in 0in 6pt;">
<p><br><span style="color: rgb(136, 136, 136);"><br>-Fab</span></p></blockquote></div>
<p> </p></div>
<p> </p></div></div></div>
<p> </p></blockquote></div></div></div></div>
<p> </p></div></div>
<div>
<p>_______________________________________________<br>ofw mailing
list<br><a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a></p>
</div></blockquote></div>
<p> </p></div>
<p> </p></div></div></div></div></div><br>_______________________________________________<br>ofw
mailing list<br><a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a><br>
</blockquote></div></div></div><br></div></div></div><br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
ofw mailing list<br>
<a href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</a><br></blockquote></div><br>