<div dir="ltr">Maybe my info is dated.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 8, 2013 at 3:16 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Good to know, this isn't being done without Oracle knowing what we are doing. We have an SR open with them, I'm just exploring options and feeding back my findings to them. They have mentioned in the past that we could use our own SM if we wanted to.</div>
<div class="gmail_extra"><div class="im"><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div class="h5"><div class="gmail_quote">On Tue, Oct 8, 2013 at 1:12 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div>AFAIK Xsigo has their own version of OpenSM with some proprietary changes that were not given back so I'm not sure all the right things will happen if you use a different version of OpenSM.</div><span><font color="#888888"><div>
</div><div>-- Hal</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 8, 2013 at 3:03 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">Yes we are running two Xsigo VP780 (now called Oracle Fabric Interconnect F-15).</div>
<div class="gmail_extra">
<div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Tue, Oct 8, 2013 at 1:00 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Are you running Xsigo hardware ?</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 8, 2013 at 1:38 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">Kevin,<div><br></div><div>Thanks for the input, I'll look into it. Does scatter ports work with MinHop, of do I need to use UpDown (the only time I've seen Scatter Ports mentioned <a href="http://www2.cisl.ucar.edu/sites/default/files/Mizero,%20F_SIParCS2013.pdf" target="_blank">http://www2.cisl.ucar.edu/sites/default/files/Mizero,%20F_SIParCS2013.pdf</a>).</div>
<div><br></div><div>Thanks,</div><div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Tue, Oct 8, 2013 at 11:35 AM, Kevin Harms <span dir="ltr"><<a href="mailto:harms@alcf.anl.gov" target="_blank">harms@alcf.anl.gov</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<br>
This is not a full solution to your problem, but you can add/set the value<br>
scatter_ports 37<br>
in your opensm.conf. This generally improves the distribution of the paths over available ports. It will not guarantee that all routes to N don't flow through a given switch but it is much more likely not to occur. The value of 37 is the initial seed for the algorithm. In our case, we did this to improve performance.<br>
<br>
kevin<br>
<div><div><br>
On Oct 8, 2013, at 12:04 PM, Robert LeBlanc <<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>> wrote:<br>
<br>
> We have been running Oracle OVN (previously Xsigo) in our data center<br>
> environment for two years now. This last week we upgraded the firmware on<br>
> our Mellanox IS5030 switches to see if that would help resolve some<br>
> communication issues for Oracle PVI and IPoIB. The Oracle OVN creates<br>
> virtual NICs and virtual HBAs and encapsulates the traffic, sends it over<br>
> the Infiniband fabric and to the directors where the data is unencapsulated<br>
> and sent out on the traditional Ethernet and Fibre Channel networks.<br>
><br>
> During our upgrade, it seems that it so happened that all four of our vHBAs<br>
> were routed through the same IS5030 switch causing all of the storage for<br>
> some of our ESX hosts to disappear when the switch was rebooted. This<br>
> caused an APD state and some of the VMs suffered corruption. We are now<br>
> looking for ways that we can make sure the routing tries to evenly spread<br>
> out all of the routes between available paths to help reduce/prevent this<br>
> in the future. We would want an algorithm that focuses on availability and<br>
> is part of the standard OFED openSM. We are looking for stability over<br>
> cutting edge. We are currently using MinHop for the routing algorithm. I'm<br>
> attaching a diagram (<br>
> <a href="https://docs.google.com/drawings/d/18pMOpiM7Bz2kaiyI0NOzB1q5o-0Zcy-9E1hJYNm6uNg/edit?usp=sharing" target="_blank">https://docs.google.com/drawings/d/18pMOpiM7Bz2kaiyI0NOzB1q5o-0Zcy-9E1hJYNm6uNg/edit?usp=sharing</a>)<br>
> of our environment as I know different algorithms are tailored for<br>
> different environments.<br>
><br>
> We are also looking to try to extract our topology, load it into IBMgtSim<br>
> and run simulations on MinHop and other algorithms to see what the<br>
> probability of having all paths run through one switch are. If you have any<br>
> pointer, we would be glad to accept them. One difficulty is that when I do<br>
> ibnetdiscover, it is showing me the ports of the HCAs, but not the node<br>
> GUID of the card. I suppose that if I see CA, I can subtract the port<br>
> number from the port GUID to get the host GUID, would that be a safe<br>
> assumption.<br>
><br>
>> ibnetdiscover<br>
> CA 56 2 0xf04da2909778e716 4x QDR - SW 52 18 0x0002c90200448e28 (<br>
> 'MT25408 ConnectX Mellanox Technologies' - 'Infiniscale-IV Mellanox<br>
> Technologies' )<br>
> CA 55 1 0xf04da2909778e715 4x QDR - SW 51 18 0x0002c90200448ec8 (<br>
> 'MT25408 ConnectX Mellanox Technologies' - 'Infiniscale-IV Mellanox<br>
> Technologies' )<br>
> ...snip...<br>
><br>
>> ibhosts<br>
> Ca : 0xf04da2909778e714 ports 2 "MT25408 ConnectX Mellanox<br>
> Technologies"<br>
> ...snip...<br>
><br>
><br>
> Thank you in advance for reading and helping us.<br>
><br>
> Robert LeBlanc<br>
> OIT Infrastructure & Virtualization Engineer<br>
> Brigham Young University<br>
</div></div>> <Fabric Design Public.pdf>_______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>