[ofa-general] Terrible TCP Qperf Performance with Centos 5.2 and default settings

[Robert Dunkley]

I see very low qperf TCP performance of IPOIB on Centos 5.2 by default
(RDMA performance is fine). 

 

Centos sets the maximum MTU for ib0 on boot automatically. For CM centos
sets the maximum MTU that is just slightly less than the logical 65536
setting. Connected mode only seems to like nice round MTU values (eg.
32768 or 2048) so the default setting always gives terrible performance.


 

Datagram mode seems to have no problem with odd MTU values like 2044
(This seems to be the max in Datagram mode). 

 

The other thing I noticed was Centos warns that multicast will be broken
with MTUs above 2044, this seems to be true, I don't need it so disabled
multicast on the Infiniband interface. 

 

Is the behaviour I'm seeing normal or is this some kind of bug? (Either
in OFED running on Centos 5.2 or QPerf) Results with various settings
below.

 

Rob

 

Default (Connected mode)

[root at mrhappy OFED-1.3.1]# qperf 192.168.10.161 tcp_bw tcp_lat

tcp_bw:

    bw  =  32.8 KB/sec

tcp_lat:

    latency  =  42.2 us

 

Datagram Mode with 2044 MTU (Max for Datagram):

[root at mrtickle OFED-1.3.1]# qperf 192.168.10.160 tcp_bw tcp_lat

tcp_bw:

    bw  =  400 MB/sec

tcp_lat:

    latency  =  41.2 us

 

Connected Mode with 32768 MTU:

[root at mrtickle ~]# qperf 192.168.10.160 tcp_bw tcp_lat

tcp_bw:

    bw  =  866 MB/sec

tcp_lat:

    latency  =  30.8 us

 


The SAQ Group
Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
SEMTEC Limited trading as SAQ is Registered in England & Wales
Company Number: 06481952

http://www.saqnet.co.uk AS29219
SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.
DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.

Find us in http://www.thebestof.co.uk/petersfield

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080725/19c11f3b/attachment.html>