[ofa-general] Re: [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank

Tue Jun 23 06:53:55 PDT 2009

Hi,

As I discussed with Line, this patch also present a problem with topolgy using reverse hops and has a negative impact on routing quality.
Moreover, I gave it some thoughts and I think there is a flaw in this approach.
For example

 L1-1    L1-2
  |       | \
  |       |  ----
  |       |      \
 _|___|___|_      L2
 | QNEM    |     |\\
 -----------     |\\\
  ////  \\\\     |\\\\
N[1-12] N[13-24] M[1-18]

In this example the horizontal link is at the last level of the tree so it is allowed by Line's patch.
As horizontal port are treated as downlinks from both sides they can only be used when creating upwards routes (algorithm going down). 
So in this example nodes N 13 to 24 will have full connectivity to nodes M 1 to 18 and nodes N 1 to 12 will have a route to nodes M 1 to 18.
However nodes M 1 to 18 won't have any routes to nodes N 1 to 12 because the algorithm starting from node N1, from example,
cannot reach switch L1-2 as the horizontal link is seen as a down link. With reverse hop it should be possible to use one horizontal link and go one but reverse hop are not well balanced and should not be used for compute nodes.

What I think should be done is have a more generalized approached. The Ftree algorithm already manages an enum with direction. We could add a 3rd direction (horizontal) and treat links as both upward and downwards. With the patch I've pushed (the one conflicting), we already ensure minimum hop path is used and secondary routes are more balanced so it should work quite nicely.

I also discussed with Line about it but I'd like to have other opinions. Why should this be restricted to max rank switches? Voltaire for example is selling 648 port switches (2x 324 with horizontal links) which could use such an update in the Fat-Tree algorithm.

Regards

Nicolas

Le 18/06/2009 15:58, Sasha Khapyorsky a écrit :
> Hi Line,
> 
> On 15:09 Tue 16 Jun     , Line.Holen at Sun.COM wrote:
>> This patch makes it legal to have cross links (horizontal links) between
>> switches at max rank. These switches do have same rank, so hop count cannot
>> be calculated based on rank anymore.
>> The horizontal links are treated as downlinks. Switch A has a downlink to B
>> while B has a downlink to A. Tests on lids and also number of hops makes sure
>> that we don't loop back and forth across the link.
>>
>> Signed-off-by: Frank Olaf Sem-Jacobsen <frankose at simula.no>
>> Signed-off-by: Line Holen <Line.Holen at sun.com>
> 
> This patch is conflicting with Nicolas's one (and his was first). Could
> you please rebase against the current master and resend. Thanks.
> 
> Sasha
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 
> 