[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

[Jim Mott]

Not today, but I will give it a shot next time I get a free machine.  I
have tested between Rhat4u4 MLX4 and Rhat4u4 mthca and seen the same
trend though.

Thanks,
JIm

Jim Mott
Mellanox Technologies Ltd.
mail: jim at mellanox.com
Phone: 512-294-5481


-----Original Message-----
From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
Sent: Friday, January 25, 2008 4:03 PM
To: Jim Mott; Weikuan Yu
Cc: general at lists.openfabrics.org
Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

Is there any way you can make sender and receiver the same RHEL kernel?

> -----Original Message-----
> From: Jim Mott [mailto:jim at mellanox.com] 
> Sent: Friday, January 25, 2008 1:58 PM
> To: Scott Weitzenkamp (sweitzen); Weikuan Yu
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP 
> performance changes inOFED 1.3 beta, and I get Oops when 
> enabling sdp_zcopy_thresh
> 
> Receive side:
>   - 2.6.23.8 kernel.org kernel on Rhat5 distro
>   - HCA is MLX4 with 2.3.914
>     I get the same number on released 2.3 firmware
> 
> Send side:
>   - 2.6.9-42.ELsmp x86_64 (Rhat4u4)
>   - HCA is MLX4 with 2.3.914
> 
> I get the same trends (SDP < BZCOPY if message_size > 64K) on 
> unmodifed
> Rhat5, Rhat4u4, and SLES10-SP1-RT distros.  I also see it on 
> kernel.org
> kernels 2.6.23.12, 2.6.24-rc2, 2.6.23, and 2.6.22.9.  I am in 
> the midst
> of testing some things, so I do not have all the machines available
> right now to repeat most of the tests though.
> 
> 
> Thanks,
> JIm
> 
> Jim Mott
> Mellanox Technologies Ltd.
> mail: jim at mellanox.com
> Phone: 512-294-5481
> 
> 
> -----Original Message-----
> From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
> Sent: Friday, January 25, 2008 3:39 PM
> To: Jim Mott; Weikuan Yu
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
> 
> Jim, what kernel and HCA are these numbers for?
> 
> Scott
> 
>  
> 
> > -----Original Message-----
> > From: Jim Mott [mailto:jim at mellanox.com] 
> > Sent: Friday, January 25, 2008 11:09 AM
> > To: Scott Weitzenkamp (sweitzen); Weikuan Yu
> > Cc: general at lists.openfabrics.org
> > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP 
> > performance changes inOFED 1.3 beta, and I get Oops when 
> > enabling sdp_zcopy_thresh
> > 
> > Right you are (as usual).
> > 
> > Hunting around these systems shows that I have been using 
> > netperf-2.4.3
> > for testing.  No configuration options; just ./configure; make; make
> > install.
> > 
> > To try and understand version differences, I installed 2.4.1 (your
> > version?), 2.4.3, and 2.4.4.  Built them with default 
> options and ran
> > the tests using each.
> > 
> > Using netperf-2.4.1 and reran "netperf -v2 -4 -H 
> > 193.168.10.143 -l 30 -t
> > TCP_STREAM -c -C --   -m size" with target AMD and driver as 
> > 8-processor
> > Intel:
> > 
> >             64K    128K      1M
> > SDP      7749.66  6925.68  6281.17
> > BZCOPY   8492.85  9867.06 11105.50
> > 
> > I tried running these tests a few times and saw a lot of 
> > variance in the
> > reported results.  Reloading 2.4.3 and running the same tests:
> > 
> >             64K    128K      1M
> > SDP      7553.77  6747.58  5986.42  
> > BZCOPY   8839.46  9572.49 10654.52
> > 
> > and finally, I tried 2.4.4 and running the same tests:
> > 
> >             64K    128K      1M
> > SDP      7935.97  6325.69  7682.65
> > BZCOPY   8905.94  9935.45 10615.03
> > 
> > At this point, I am confused.  The difference between SDP with and
> > without Bzcopy is obvious in all three sets of numbers.  I can not
> > explain why you see something different.  
> > 
> > If you could try a vanilla netperf build, it would be 
> > interesting to see
> > if you get any different results.
> > 
> > Thanks,
> > JIm
> > 
> > Jim Mott
> > Mellanox Technologies Ltd.
> > mail: jim at mellanox.com
> > Phone: 512-294-5481
> > 
> > 
> > -----Original Message-----
> > From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
> > Sent: Friday, January 25, 2008 10:36 AM
> > To: Jim Mott; Jim Mott; Weikuan Yu
> > Cc: general at lists.openfabrics.org
> > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
> > changes inOFED 1.3 beta, and I get Oops when enabling 
> sdp_zcopy_thresh
> > 
> > > So I see your results (sort of).  I have been using the 
> > > netperf that ships with the OS (Rhat4u4 and Rhat5 mostly) or 
> > > is built with
> > > default options.  Maybe that is the difference.
> > 
> > Jim, AFAIK Red Hat does not ship netperf with RHEL.
> > 
> > Scott
> > 
>