[openib-general] A few questions about IBMgtSim

[Sven-Arne Reinemo]

Anno Domini 22-07-2006 20:25, Eitan Zahavi wrote:
> Hi Sven,
> 
>>> Currently there is no way to scale simulation time to real time. 
>>> The main reason is that the time scale is mixed: * OpenSM
>>> calculation time is about the same (if you run the simulator on
>>> remote node)
>> So this means that the internal operation of OpenSM with the
>> simulator is identical to its operation with real hardware?
> [EZ] Yes, if the algorithmic stage is only computational (like the
> routing stage) the time it takes is the sane as real hardware. But
> the entire fabric setting is involving sending and receiving MADs
> thus odes not scale.
>> I have done some performance test with IBMgtSim and OpenSM running
>> on separate machines and to me it looks like there is very little
>> concurrency between the two processes. I.e. it looks like they
>> spend a lot of time  waiting for each other. Below are some results
>> from a few simulation runs, the observed CPU utilization seems
>> quite low. I would have expected much higher CPU load for
>> IBMgtSim...  Any thoughts on how this matches your experience?
> [EZ] Yes - these is not much concurrency. Actually it really depends
> on the number of MADs you allow on the wire. Also, one of the major
> limitations I run into (which made me split the processes to 2
> machine) was memory availability for the 10K nodes case.

Is it possible to configure the number of MADs allowd on the wire?

> 
> I do not see what is the drive for doing these comparisons. BTW: do
> you plan to run the OpenSM tests over the simulator?

Yes. As start we would like to experiment with OpenSM and possible
enhancements by using the simulator. We have compared IBMgtSim with some
of our own simulation tools. Our tools all do flit level simulations so
they do not scale very well. Neither do they support IBA management
traffic. When doing these test it was natural to look for benefits of
distributing the process (apart from the  memory requirements).

> 
>> OpenSM #hosts² #sw #ports elapsed¹ kernel¹ user¹ %cpu mem 288   36
>> 24     585    109   99   35 410 512   48     32     766    144  136
>> 36 520 1152   72     48    1161    218  211   36 741
>> 
>> IBMgtSim #hosts² #sw #ports elapsed¹ kernel¹ user¹ %cpu mem 288
>> 36     24     586      87   221   52  92 512   48     32     767
>> 109   278   50 102 1152   72     48    1161     169   432   51 132
>> 
>> ¹time in seconds ²organized in a 3 stage Clos
>> 
>> Best regards, Sven-Arne
>> 
>> -- SAR ---- GnuPG public key -
>> http://home.ifi.uio.no/~svenar/gpg.asc ----  "There are only 10
>> kinds of people in this world; those who know binary and those who
>> don't." -- Unknown



-- 
SAR
---- GnuPG public key - http://home.ifi.uio.no/~svenar/gpg.asc ----
 "There are only 10 kinds of people in this world; those who know
  binary and those who don't."
                                           -- Unknown