[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

[Jim Mott]

Receive side:
  - 2.6.23.8 kernel.org kernel on Rhat5 distro
  - HCA is MLX4 with 2.3.914
    I get the same number on released 2.3 firmware

Send side:
  - 2.6.9-42.ELsmp x86_64 (Rhat4u4)
  - HCA is MLX4 with 2.3.914

I get the same trends (SDP < BZCOPY if message_size > 64K) on unmodifed
Rhat5, Rhat4u4, and SLES10-SP1-RT distros.  I also see it on kernel.org
kernels 2.6.23.12, 2.6.24-rc2, 2.6.23, and 2.6.22.9.  I am in the midst
of testing some things, so I do not have all the machines available
right now to repeat most of the tests though.


Thanks,
JIm

Jim Mott
Mellanox Technologies Ltd.
mail: jim at mellanox.com
Phone: 512-294-5481


-----Original Message-----
From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
Sent: Friday, January 25, 2008 3:39 PM
To: Jim Mott; Weikuan Yu
Cc: general at lists.openfabrics.org
Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

Jim, what kernel and HCA are these numbers for?

Scott

 

> -----Original Message-----
> From: Jim Mott [mailto:jim at mellanox.com] 
> Sent: Friday, January 25, 2008 11:09 AM
> To: Scott Weitzenkamp (sweitzen); Weikuan Yu
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP 
> performance changes inOFED 1.3 beta, and I get Oops when 
> enabling sdp_zcopy_thresh
> 
> Right you are (as usual).
> 
> Hunting around these systems shows that I have been using 
> netperf-2.4.3
> for testing.  No configuration options; just ./configure; make; make
> install.
> 
> To try and understand version differences, I installed 2.4.1 (your
> version?), 2.4.3, and 2.4.4.  Built them with default options and ran
> the tests using each.
> 
> Using netperf-2.4.1 and reran "netperf -v2 -4 -H 
> 193.168.10.143 -l 30 -t
> TCP_STREAM -c -C --   -m size" with target AMD and driver as 
> 8-processor
> Intel:
> 
>             64K    128K      1M
> SDP      7749.66  6925.68  6281.17
> BZCOPY   8492.85  9867.06 11105.50
> 
> I tried running these tests a few times and saw a lot of 
> variance in the
> reported results.  Reloading 2.4.3 and running the same tests:
> 
>             64K    128K      1M
> SDP      7553.77  6747.58  5986.42  
> BZCOPY   8839.46  9572.49 10654.52
> 
> and finally, I tried 2.4.4 and running the same tests:
> 
>             64K    128K      1M
> SDP      7935.97  6325.69  7682.65
> BZCOPY   8905.94  9935.45 10615.03
> 
> At this point, I am confused.  The difference between SDP with and
> without Bzcopy is obvious in all three sets of numbers.  I can not
> explain why you see something different.  
> 
> If you could try a vanilla netperf build, it would be 
> interesting to see
> if you get any different results.
> 
> Thanks,
> JIm
> 
> Jim Mott
> Mellanox Technologies Ltd.
> mail: jim at mellanox.com
> Phone: 512-294-5481
> 
> 
> -----Original Message-----
> From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
> Sent: Friday, January 25, 2008 10:36 AM
> To: Jim Mott; Jim Mott; Weikuan Yu
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
> 
> > So I see your results (sort of).  I have been using the 
> > netperf that ships with the OS (Rhat4u4 and Rhat5 mostly) or 
> > is built with
> > default options.  Maybe that is the difference.
> 
> Jim, AFAIK Red Hat does not ship netperf with RHEL.
> 
> Scott
>