***SPAM*** Re: [ofa-general] Terrible TCP Qperf Performance with Centos 5.2 and default settings

[Dotan Barak]

Robert Dunkley wrote:
>
> I see very low qperf TCP performance of IPOIB on Centos 5.2 by default 
> (RDMA performance is fine).
>
Which HW do you use IB or IWarp?

thanks
Dotan
>
> Centos sets the maximum MTU for ib0 on boot automatically. For CM 
> centos sets the maximum MTU that is just slightly less than the 
> logical 65536 setting. Connected mode only seems to like nice round 
> MTU values (eg. 32768 or 2048) so the default setting always gives 
> terrible performance.
>
> Datagram mode seems to have no problem with odd MTU values like 2044 
> (This seems to be the max in Datagram mode).
>
> The other thing I noticed was Centos warns that multicast will be 
> broken with MTUs above 2044, this seems to be true, I don’t need it so 
> disabled multicast on the Infiniband interface.
>
> Is the behaviour I’m seeing normal or is this some kind of bug? 
> (Either in OFED running on Centos 5.2 or QPerf) Results with various 
> settings below.
>
> Rob
>
> Default (Connected mode)
>
> [root at mrhappy OFED-1.3.1]# qperf 192.168.10.161 tcp_bw tcp_lat
>
> tcp_bw:
>
> bw = 32.8 KB/sec
>
> tcp_lat:
>
> latency = 42.2 us
>
> Datagram Mode with 2044 MTU (Max for Datagram):
>
> [root at mrtickle OFED-1.3.1]# qperf 192.168.10.160 tcp_bw tcp_lat
>
> tcp_bw:
>
> bw = 400 MB/sec
>
> tcp_lat:
>
> latency = 41.2 us
>
> Connected Mode with 32768 MTU:
>
> [root at mrtickle ~]# qperf 192.168.10.160 tcp_bw tcp_lat
>
> tcp_bw:
>
> bw = 866 MB/sec
>
> tcp_lat:
>
> latency = 30.8 us
>
> **
>
> **
>
> *The SAQ Group*
>
> *Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ*
> SEMTEC Limited Trading as SAQ is Registered in England & Wales
> Company Number: 06481952
>
> http://www.saqnet.co.uk <http://www.saqnet.co.uk/> AS29219
>
> SAQ Group Delivers high quality, honestly priced communication and 
> I.T. services to UK Business.
>
> DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : 
> Backups : Managed Networks : Remote Support.
>
> Find us in http://www.thebestof.co.uk/petersfield
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general