[ofa-general] opensm: Routing on non-pure Fat-Tree

[Yevgeny Kliteynik]

Nicolas Morey Chaisemartin wrote:
> Yevgeny Kliteynik a écrit :
>> Hi Nicolas,
>>
>> Nicolas Morey Chaisemartin wrote:
>>> Hello,
>>>
>>> I am conducting some tests on routing non-pure fat-tree network using 
>>> the fat tree algorithm of OpenSM.
>>> The network I am experimenting on is a 3 level fat tree, with a 
>>> pruned 3rd layer.
>>> By providing the root_guid_file, the algorithm works great !
>>>
>>> The problem is, we would like to add some service nodes directly on 
>>> the 3rd level switches.
>>> I have added the cn_guid_file so the network is still recognize as a 
>>> fat tree.
>>> OpenSM once more manage to create the routing for the network. It 
>>> provides full connectivity,
>>> except there are no routes between non computes nodes.
>>> I understand that the point of setting these node as not compute node 
>>> should intend they won't talk to each other, but we still need a bit 
>>> of connectivity between them to exchange few datas (pings and such).
>>> A simple min-hop or such should be enough to generate those routes.
>>> It will probably desequilibrate the number of routes going through 
>>> the top links, but those additional link makes virtually no traffic 
>>> at all, so in practical it shouldn't be a problem.
>>
>> Fat-tree should create full connectivity as long as there is an up/down
>> route between ports. Do you get connectivity between these nodes with
>> up/down routing algorithm?
>> Try running it with the same root_guid_file.
>>
>> -- Yevgeny
> 
> Well the route would be more down/up compared to the rest of the transfer.
> (Im not sure I was clear, but when i talk of 3rd level, I mean top 
> level. 1st level begin the switches just above the compute nodes)

Oh, OK. I was thinking the opposite. So you connect these
non-CNs to spine switches.

> I'll try this tomorrow

No need :)
Fat-tree is a variation of up/down routing. As such, down/up
routes are not allowed. You won't have connectivity between
these nodes neither in fat-tree nor in up/down routing.

>>> Is there any reasons such a behavior wasn't implemented yet?

The idea of allowing only up/down routes is preventing credit
loops in the fabric.

>>> Should there be one?

I guess it is possible, but these down/up routes will create
credit loops, so any traffic between these "special" nodes is
potentially bad for fabric.
Note that there's already a "connect roots" option in the
up/down routing which violates the up/down rule, but this is
only between switches, so I believe that the only traffic
that uses these routes is management traffic.

-- Yevgeny

>>> Regards
>>>
>>> Nicolas Morey-Chaisemartin
>>>
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit 
>>> http://openib.org/mailman/listinfo/openib-general
>>>
>>
>>
>>
> 
>