[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

[Weikuan Yu]

Hi, Jim and Scott,

Just to provide some additional information, I have seen no performance 
improvement either. I tried both a pair of old 32-bit Xeons, and a pair 
of woodcrest 5100. I used a recent kernel (2.6.23.14) and the nightly 
tarball OFED-1.3-rc4.

My HCAs were running in Tavor modes though. I am in the process of 
updating firmware and trying for connectX.

--Weikuan

Jim Mott wrote:
> Not today, but I will give it a shot next time I get a free machine.  I
> have tested between Rhat4u4 MLX4 and Rhat4u4 mthca and seen the same
> trend though.
> 
> Thanks,
> JIm
> 
> Jim Mott
> Mellanox Technologies Ltd.
> mail: jim at mellanox.com
> Phone: 512-294-5481
> 
> 
> -----Original Message-----
> From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
> Sent: Friday, January 25, 2008 4:03 PM
> To: Jim Mott; Weikuan Yu
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
> 
> Is there any way you can make sender and receiver the same RHEL kernel?
> 
>> -----Original Message-----
>> From: Jim Mott [mailto:jim at mellanox.com] 
>> Sent: Friday, January 25, 2008 1:58 PM
>> To: Scott Weitzenkamp (sweitzen); Weikuan Yu
>> Cc: general at lists.openfabrics.org
>> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP 
>> performance changes inOFED 1.3 beta, and I get Oops when 
>> enabling sdp_zcopy_thresh
>>
>> Receive side:
>>   - 2.6.23.8 kernel.org kernel on Rhat5 distro
>>   - HCA is MLX4 with 2.3.914
>>     I get the same number on released 2.3 firmware
>>
>> Send side:
>>   - 2.6.9-42.ELsmp x86_64 (Rhat4u4)
>>   - HCA is MLX4 with 2.3.914
>>
>> I get the same trends (SDP < BZCOPY if message_size > 64K) on 
>> unmodifed
>> Rhat5, Rhat4u4, and SLES10-SP1-RT distros.  I also see it on 
>> kernel.org
>> kernels 2.6.23.12, 2.6.24-rc2, 2.6.23, and 2.6.22.9.  I am in 
>> the midst
>> of testing some things, so I do not have all the machines available
>> right now to repeat most of the tests though.
>>
>>
>> Thanks,
>> JIm
>>
>> Jim Mott
>> Mellanox Technologies Ltd.
>> mail: jim at mellanox.com
>> Phone: 512-294-5481
>>
>>
>> -----Original Message-----
>> From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
>> Sent: Friday, January 25, 2008 3:39 PM
>> To: Jim Mott; Weikuan Yu
>> Cc: general at lists.openfabrics.org
>> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
>> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
>>
>> Jim, what kernel and HCA are these numbers for?
>>
>> Scott
>>
>>  
>>
>>> -----Original Message-----
>>> From: Jim Mott [mailto:jim at mellanox.com] 
>>> Sent: Friday, January 25, 2008 11:09 AM
>>> To: Scott Weitzenkamp (sweitzen); Weikuan Yu
>>> Cc: general at lists.openfabrics.org
>>> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP 
>>> performance changes inOFED 1.3 beta, and I get Oops when 
>>> enabling sdp_zcopy_thresh
>>>
>>> Right you are (as usual).
>>>
>>> Hunting around these systems shows that I have been using 
>>> netperf-2.4.3
>>> for testing.  No configuration options; just ./configure; make; make
>>> install.
>>>
>>> To try and understand version differences, I installed 2.4.1 (your
>>> version?), 2.4.3, and 2.4.4.  Built them with default 
>> options and ran
>>> the tests using each.
>>>
>>> Using netperf-2.4.1 and reran "netperf -v2 -4 -H 
>>> 193.168.10.143 -l 30 -t
>>> TCP_STREAM -c -C --   -m size" with target AMD and driver as 
>>> 8-processor
>>> Intel:
>>>
>>>             64K    128K      1M
>>> SDP      7749.66  6925.68  6281.17
>>> BZCOPY   8492.85  9867.06 11105.50
>>>
>>> I tried running these tests a few times and saw a lot of 
>>> variance in the
>>> reported results.  Reloading 2.4.3 and running the same tests:
>>>
>>>             64K    128K      1M
>>> SDP      7553.77  6747.58  5986.42  
>>> BZCOPY   8839.46  9572.49 10654.52
>>>
>>> and finally, I tried 2.4.4 and running the same tests:
>>>
>>>             64K    128K      1M
>>> SDP      7935.97  6325.69  7682.65
>>> BZCOPY   8905.94  9935.45 10615.03
>>>
>>> At this point, I am confused.  The difference between SDP with and
>>> without Bzcopy is obvious in all three sets of numbers.  I can not
>>> explain why you see something different.  
>>>
>>> If you could try a vanilla netperf build, it would be 
>>> interesting to see
>>> if you get any different results.
>>>
>>> Thanks,
>>> JIm
>>>
>>> Jim Mott
>>> Mellanox Technologies Ltd.
>>> mail: jim at mellanox.com
>>> Phone: 512-294-5481
>>>
>>>
>>> -----Original Message-----
>>> From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
>>> Sent: Friday, January 25, 2008 10:36 AM
>>> To: Jim Mott; Jim Mott; Weikuan Yu
>>> Cc: general at lists.openfabrics.org
>>> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
>>> changes inOFED 1.3 beta, and I get Oops when enabling 
>> sdp_zcopy_thresh
>>>> So I see your results (sort of).  I have been using the 
>>>> netperf that ships with the OS (Rhat4u4 and Rhat5 mostly) or 
>>>> is built with
>>>> default options.  Maybe that is the difference.
>>> Jim, AFAIK Red Hat does not ship netperf with RHEL.
>>>
>>> Scott
>>>
>