[openib-general] A few questions about IBMgtSim

[Hal Rosenstock]

On Tue, 2006-08-01 at 07:04, Sven-Arne Reinemo wrote:
> Anno Domini 22-07-2006 20:25, Eitan Zahavi wrote:
> > Hi Sven,
> > 
> >>> Currently there is no way to scale simulation time to real time. 
> >>> The main reason is that the time scale is mixed: * OpenSM
> >>> calculation time is about the same (if you run the simulator on
> >>> remote node)
> >> So this means that the internal operation of OpenSM with the
> >> simulator is identical to its operation with real hardware?
> > [EZ] Yes, if the algorithmic stage is only computational (like the
> > routing stage) the time it takes is the sane as real hardware. But
> > the entire fabric setting is involving sending and receiving MADs
> > thus odes not scale.
> >> I have done some performance test with IBMgtSim and OpenSM running
> >> on separate machines and to me it looks like there is very little
> >> concurrency between the two processes. I.e. it looks like they
> >> spend a lot of time  waiting for each other. Below are some results
> >> from a few simulation runs, the observed CPU utilization seems
> >> quite low. I would have expected much higher CPU load for
> >> IBMgtSim...  Any thoughts on how this matches your experience?
> > [EZ] Yes - these is not much concurrency. Actually it really depends
> > on the number of MADs you allow on the wire. Also, one of the major
> > limitations I run into (which made me split the processes to 2
> > machine) was memory availability for the 10K nodes case.
> 
> Is it possible to configure the number of MADs allowd on the wire?

-maxsmps option to OpenSM or 
-maxsmps <number>
          This option specifies the number of VL15 SMP MADs
          allowed on the wire at any one time.
          Specifying -maxsmps 0 allows unlimited outstanding
          SMPs.
          Without -maxsmps, OpenSM defaults to a maximum of
          4 outstanding SMPs.

or in /var/cache/osm/opensm.opts:

# Number of MADs sent in parallel
max_wire_smps 4

-- Hal

> > 
> > I do not see what is the drive for doing these comparisons. BTW: do
> > you plan to run the OpenSM tests over the simulator?
> 
> Yes. As start we would like to experiment with OpenSM and possible
> enhancements by using the simulator. We have compared IBMgtSim with some
> of our own simulation tools. Our tools all do flit level simulations so
> they do not scale very well. Neither do they support IBA management
> traffic. When doing these test it was natural to look for benefits of
> distributing the process (apart from the  memory requirements).
> 
> > 
> >> OpenSM #hosts² #sw #ports elapsed¹ kernel¹ user¹ %cpu mem 288   36
> >> 24     585    109   99   35 410 512   48     32     766    144  136
> >> 36 520 1152   72     48    1161    218  211   36 741
> >> 
> >> IBMgtSim #hosts² #sw #ports elapsed¹ kernel¹ user¹ %cpu mem 288
> >> 36     24     586      87   221   52  92 512   48     32     767
> >> 109   278   50 102 1152   72     48    1161     169   432   51 132
> >> 
> >> ¹time in seconds ²organized in a 3 stage Clos
> >> 
> >> Best regards, Sven-Arne
> >> 
> >> -- SAR ---- GnuPG public key -
> >> http://home.ifi.uio.no/~svenar/gpg.asc ----  "There are only 10
> >> kinds of people in this world; those who know binary and those who
> >> don't." -- Unknown
> 
>