[ofa-general] SDP performance with bzcopy testing help needed

[Jim Mott]

Now that SDP is shipping with a non-zero default value for
sdp_zcopy_thresh (64K), I need some feedback from the list.  Does
anybody except me see a performance gain on large messages?

1)  nerperf server node
      - export LD_PRELOAD=libsdp.so
      - netserver

2) netperf client node
      - export LD_PRELOAD=libsdp.so

3) Run some tests with bzcopy enabled on client
      - cat /sys/module/ib_sdp/sdp_zcopy_thresh
        65536
        (You might have to hunt for it in parameters)

      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m
64K
      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m
128K
      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m 1M

4) Run the tests with bzcopy disabled on the client
      - echo 0 > /sys/module/ib_sdp/sdp_zcopy_thresh

      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m
64K
      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m
128K
      - netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---m 1M


When I run these tests on ConnectX DDR through a switch, I see bandwidth
numbers that look like:

              64K    128K      1M
  SDP      8215.17  6429.09  6862.66
  BZCOPY   8748.00  9997.07  9847.76

Looking at uS/KB transferred we see:

               64K         128K          1M
            LCL   RMT    LCL   RMT    LCL   RMT
  SDP      1.025 1.243  1.391 1.493  1.274 1.407
  BZCOPY   0.966 1.148  0.838 1.014  0.603 0.984

Note that it is important that you use "-m" instead of "-r" because you
want to present the largest possible buffers (netperf "Send Message
Size") to SDP.  Pinning user memory instead of copying through the
kernel is only a win if you can amortize the costs over a big chunk of
memory.