[ofa-general] Couple of questions about OpenSM

[Sasha Khapyorsky]

Hi Nicolas,

On 11:12 Thu 19 Mar     , Nicolas Morey Chaisemartin wrote:
> 
> First question and an easy one:
> By default with what optimization options is OpenSM compiled with? 

It is defined in autoconf I guess.

> Without any specific options using git, there a no -O2 or such so all the inline functions are not inline which make a huge bad impact on performances (few millions calls to osm_switch_get_least_hops not inline consumes over 15% of computing time)

In my setup I have in /usr/share/autoconf/autoconf/c.m4:

  if test "$GCC" = yes; then
    CFLAGS="-g -O2"

And -O2 is turned on by default.

> Next one and a bit harder:
> In the Fat-Tree we have a 2D array for hop table (destination lid/port num). Why is this table allocated as we need and not all at once?

Yes, I think it can be preallocated. Just not that this should not have
port arrays for each lid in a fabric, but only for switches.

> And is it really necessary to check  each time if the lid we use is not greater than max_lid_ho ?

May be not, but I will need to check carefully.

> The only reason I would see for this is if a new node/switch with a bigger lid was added to the fabric while openSM is routing. In such a case, wouldn't a lock protect the variables so new lid can't appear/disappear while it calculates the routes ?

Routing calculation cannot happen in parallel with discovery (it is all
serialized in do_sweep() function), we should be protected at least in
this part.

> If yes, we could allocate all and skip a lot of checks. We have millions of calls to malloc and memset in osm_switch_set_hops plus tests in get_hops/get_least_hops.

malloc() calls are conditional, there could be many checks, but only
"needed" amount of malloc() itself.

> This may cost a bit more memory,

Min hop's port arrays preallocation should not cost any extra memory (if
done properly - for switches only) - we are allocating all needed buffers
in routing calculation time anyway.

> but easily gain 15% on routing computation time.

Well, I'm skeptical about 15% :). But it doesn't really matter - even
1% performance gain and/or cleaner code would be nice improvement.

Sasha