[ofa-general] Re: [PATCH OpenSM 0/3] Fat Tree - Routing between non-CN nodes

[Nicolas Morey-Chaisemartin]

Sasha Khapyorsky a écrit :
> Hi Nicolas,
>
> On 14:42 Thu 18 Dec     , Nicolas Morey Chaisemartin wrote:
>   
>> We are current working on a Ftree topology where IO nodes are connected on 
>> spine switches.
>> Using the cn_guid_file and root_guid_file works great.
>> It is possible to route the whole tree as a fat tree. All the CNs are 
>> connected to the other CN and IO nodes.
>> However, we are missing some connectivity between IO nodes. This is the 
>> expected behavior as the route between those IO nodes would have
>> to go down to go back up on another spine switch.
>>
>> However, we need at least a bit of connectivity between those nodes. There 
>> won't be any real traffic but just some "ping" for HA purposes.
>>
>> Therefore, I have implemented two new options to openSM: io_guid_file and 
>> max_reverse_hops.
>> The io_guid_file provides a list of all the IO guid (it may differs from 
>> the list of non-CN nodes)
>>     
>
> "IO" is specific for your setup. Could we find more generic name for such
> nodes?
>
>   
Sure. Any ideas?
>> The max_reverse_hops gives the number of time IO nodes (described by 
>> io_guid_file) are allowed to use a switch backward.
>>     
>
> Don't those two options duplicate each others somehow? If we want to
> connect io nodes anyway, why max_reverse_hops should be important?
>   
Because we may not want to connect all of them to all the nodes. By
specifying a small max_reverse_hop you can restrain (depending on your
topology) the effect of the io_guid_file so an "IO" node will only see
the closests "IO" node through reverse routes but not all of them
As the effect on credit loop is not certain yet, I think the less
reverse route we create, the better it is.
> Or probably instead of having io nodes guids list we prefer to connect
> everything N hops from roots? Then sort of --connect-roots extension
> (--connect-roots=3) could work. No?
>
>   
That should work too but it is less flexible than io_guid_file for
tweaking the configuration and have the best routing scheme.
>> According to my tests this has absolutely no effects on regular routing and 
>> manages to connect the io nodes together, if max_reverse_hops is big 
>> enough.
>>
>> This is a first draft for this feature. I'd be happy to have some feedback 
>> about how to upgrade it and make it as clean as possible, wether it is 
>> integrated in the mainstream or not.
>>     
>
> Since this functionality is optional, useful and shouldn't change a
> default behavior it can be suitable for main stream IMO.
>
> Sasha
Okay, I'll fix the indentation and few coding style error. There is also
a bug in the current patch as the hops counter are not set to the right
value when creating route which had reverse hops number of reverse
hops*2 should be added.

I'll have to rewrite the patches so they work with the current HEAD.
Specially with the option system changes it won't merge cleanly.

Nicolas