[ofa-general] SDP performance with bzcopy testing help needed

[Craig Prescott]

Craig Prescott wrote:
> Scott Weitzenkamp (sweitzen) wrote:
>>> But the effect is still clear.
>>>
>>> throughput:
>>>
>>>                64K    128K      1M
>>>    SDP      7602.40  7560.57  5791.56
>>>    BZCOPY   5454.20  6378.48  7316.28
>> Looks unclear to me.  Sometimes BZCOPY does better, sometimes worse.   
> Fair enough.
>
> While measuring a broader spectrum of message sizes, I noted a
> big variation in throughput and send service demand for the SDP
> <snip>
Sorry for the slow follow up.  I've tried to get to the bottom of this.
I think what I've found could explain the mixed results that have been
observed when people try to see any benefit from BZCOPY. 

In a nutshell, on my setup I found that normal SDP "Bcopy" throughput
and send service demand is highly dependent upon which core the
netperf client runs on (is it the same core handling ib_mthca
interrupts?) and whether or not the pages mapped by the netperf
client are on a node local to the core executing that client.

BZCOPY, on the other hand, shows only weak dependence upon these
variables.

What I did was the following:

1) Fix the netserver and ib_mthca interrupt mappings on specific
   cores. 
2) On the netperf client machine, run netperf on each core for
   message sizes from 64KiB to 4MiB serially while holding
   ib_mthca interrupt mappings constant.
3) Repeat step 2) for all possible ib_mthca interrupt mappings.
4) Repeat step 2) and 3) with the client booted with "mem=1024M".

Step 4) forces all pages mapped by the netperf client to come
from node 0 (can see with /proc/<pid>/numa_maps).

On my setup, BZCOPY performance began to be competitive with
Bcopy once the message sizes reached about 256KiB.  Somewhere
between 1MiB and 2MiB, BZCOPY send service demand was *always*
less than Bcopy - even under the most advantageous conditions
for Bcopy.

I documented the setup and results here:

http://hpc.ufl.edu/benchmarks/ib_sdp/smp_aff.html

The last plots on the page are the ones to look at if you
don't want to read the whole thing.

So there is a class of conditions for which SDP Bcopy will
remain competitve with BZCOPY, even as message sizes
become quite large.  I think this may explain why some
people see it, and some people don't.

Cheers,
Craig