[ofa-general] ipoib performanec at smaller mtu

[Ben]

here is the iperf test:

[ ID] Interval       Transfer     Bandwidth
[  9]  0.0-10.0 sec    244 MBytes    205 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec    222 MBytes    187 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 10]  0.0-10.0 sec    190 MBytes    160 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec    406 MBytes    340 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec    204 MBytes    171 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec    229 MBytes    191 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    222 MBytes    186 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  7]  0.0-10.6 sec    147 MBytes    116 Mbits/sec
[SUM]  0.0-10.6 sec  1.82 GBytes  1.47 Gbits/sec

On a RHEL5 with OFED 1.3 machine with similar ib cards, I can get  
about 3x this.

-Ben

On Mar 10, 2009, at 6:20 PM, Boris Shpolyansky wrote:

> Ben,
>
> What is the 8-stream BW you are seeing with IPoIB when using datagram
> mode and MTU=1500?
>
> Boris Shpolyansky
> Sr. Member of Technical Staff, Applications
>
> Mellanox Technologies Inc.
> 350 Oakmead Parkway, Suite 100
> Sunnyvale, CA 94085
> Tel.: (408) 916 0014
> Fax: (408) 585 0314
> Cell: (408) 834 9365
> www.mellanox.com
>
> -----Original Message-----
> From: Ben [mailto:ben at lanl.gov]
> Sent: Monday, March 09, 2009 5:19 PM
> To: Boris Shpolyansky
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] ipoib performanec at smaller mtu
>
> I have a similar setup in datagram, and don't see a big difference.
> Are these numbers expected?  I guess I would like to understand the
> limitations.  It seams like these numbers are a bit below what the
> hardware is capable of even with lro off.
>
> -Ben
>
> On Mar 9, 2009, at 6:09 PM, Boris Shpolyansky wrote:
>
>> I think for small MTU sizes you will see better performance with  
>> IPoIB
>> in datagram mode.
>>
>> Boris Shpolyansky
>> Sr. Member of Technical Staff, Applications
>>
>> Mellanox Technologies Inc.
>> 350 Oakmead Parkway, Suite 100
>> Sunnyvale, CA 94085
>> Tel.: (408) 916 0014
>> Fax: (408) 585 0314
>> Cell: (408) 834 9365
>> www.mellanox.com
>>
>> -----Original Message-----
>> From: general-bounces at lists.openfabrics.org
>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of
>> Benjamin M.
>> McClelland
>> Sent: Monday, March 09, 2009 4:19 PM
>> To: general at lists.openfabrics.org
>> Subject: [ofa-general] ipoib performanec at smaller mtu
>>
>> I was curious what other people are seeing for 1500 mtu  
>> performance on
>> ipoib.  Current tests are showing that I get about 150-175MB/s no
>> matter
>> how many streams.  If I go up to higher mtu, then performance
>> significantly increases.  Due to the storage network that this ib is
>> routing to, we are currently limited to 1500 mtu.
>>
>> ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox
>> Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw
>> 2.6.000, voltaire sm on switch
>>
>> lro disabled, send_queue_size=256
>>
>> thanks,
>> -Ben
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>