[Users] Subnet question

[Robert LeBlanc]

Yes we are running two Xsigo VP780 (now called Oracle Fabric Interconnect
F-15).


Robert LeBlanc
OIT Infrastructure & Virtualization Engineer
Brigham Young University


On Tue, Oct 8, 2013 at 1:00 PM, Hal Rosenstock <hal.rosenstock at gmail.com>wrote:

> Are you running Xsigo hardware ?
>
>
> On Tue, Oct 8, 2013 at 1:38 PM, Robert LeBlanc <robert_leblanc at byu.edu>wrote:
>
>> Kevin,
>>
>> Thanks for the input, I'll look into it. Does scatter ports work with
>> MinHop, of do I need to use UpDown (the only time I've seen Scatter Ports
>> mentioned
>> http://www2.cisl.ucar.edu/sites/default/files/Mizero,%20F_SIParCS2013.pdf
>> ).
>>
>> Thanks,
>>
>>
>> Robert LeBlanc
>> OIT Infrastructure & Virtualization Engineer
>> Brigham Young University
>>
>>
>> On Tue, Oct 8, 2013 at 11:35 AM, Kevin Harms <harms at alcf.anl.gov> wrote:
>>
>>>
>>>   This is not a full solution to your problem, but you can add/set the
>>> value
>>>   scatter_ports 37
>>>   in your opensm.conf. This generally improves the distribution of the
>>> paths over available ports. It will not guarantee that all routes to N
>>> don't flow through a given switch but it is much more likely not to occur.
>>> The value of 37 is the initial seed for the algorithm. In our case, we did
>>> this to improve performance.
>>>
>>> kevin
>>>
>>> On Oct 8, 2013, at 12:04 PM, Robert LeBlanc <robert_leblanc at byu.edu>
>>> wrote:
>>>
>>> > We have been running Oracle OVN (previously Xsigo) in our data center
>>> > environment for two years now. This last week we upgraded the firmware
>>> on
>>> > our Mellanox IS5030 switches to see if that would help resolve some
>>> > communication issues for Oracle PVI and IPoIB. The Oracle OVN creates
>>> > virtual NICs and virtual HBAs and encapsulates the traffic, sends it
>>> over
>>> > the Infiniband fabric and to the directors where the data is
>>> unencapsulated
>>> > and sent out on the traditional Ethernet and Fibre Channel networks.
>>> >
>>> > During our upgrade, it seems that it so happened that all four of our
>>> vHBAs
>>> > were routed through the same IS5030 switch causing all of the storage
>>> for
>>> > some of our ESX hosts to disappear when the switch was rebooted. This
>>> > caused an APD state and some of the VMs suffered corruption. We are now
>>> > looking for ways that we can make sure the routing tries to evenly
>>> spread
>>> > out all of the routes between available paths to help reduce/prevent
>>> this
>>> > in the future. We would want an algorithm that focuses on availability
>>> and
>>> > is part of the standard OFED openSM. We are looking for stability over
>>> > cutting edge. We are currently using MinHop for the routing algorithm.
>>> I'm
>>> > attaching a diagram (
>>> >
>>> https://docs.google.com/drawings/d/18pMOpiM7Bz2kaiyI0NOzB1q5o-0Zcy-9E1hJYNm6uNg/edit?usp=sharing
>>> )
>>> > of our environment as I know different algorithms are tailored for
>>> > different environments.
>>> >
>>> > We are also looking to try to extract our topology, load it into
>>> IBMgtSim
>>> > and run simulations on MinHop and other algorithms to see what the
>>> > probability of having all paths run through one switch are. If you
>>> have any
>>> > pointer, we would be glad to accept them. One difficulty is that when
>>> I do
>>> > ibnetdiscover, it is showing me the ports of the HCAs, but not the node
>>> > GUID of the card. I suppose that if I see CA, I can subtract the port
>>> > number from the port GUID to get the host GUID, would that be a safe
>>> > assumption.
>>> >
>>> >> ibnetdiscover
>>> > CA    56  2 0xf04da2909778e716 4x QDR - SW    52 18 0x0002c90200448e28
>>> (
>>> > 'MT25408 ConnectX Mellanox Technologies' - 'Infiniscale-IV Mellanox
>>> > Technologies' )
>>> > CA    55  1 0xf04da2909778e715 4x QDR - SW    51 18 0x0002c90200448ec8
>>> (
>>> > 'MT25408 ConnectX Mellanox Technologies' - 'Infiniscale-IV Mellanox
>>> > Technologies' )
>>> > ...snip...
>>> >
>>> >> ibhosts
>>> > Ca      : 0xf04da2909778e714 ports 2 "MT25408 ConnectX Mellanox
>>> > Technologies"
>>> > ...snip...
>>> >
>>> >
>>> > Thank you in advance for reading and helping us.
>>> >
>>> > Robert LeBlanc
>>> > OIT Infrastructure & Virtualization Engineer
>>> > Brigham Young University
>>> > <Fabric Design
>>> Public.pdf>_______________________________________________
>>> > Users mailing list
>>> > Users at lists.openfabrics.org
>>> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20131008/7ee6c361/attachment.html>