[ofa-general] Direct / "Crossover" IPOIB

[Robert Dunkley]

Just a general question and I apologise if my questions seem simplistic.


I have played with the larger MTU settings in connected mode (32768
seems to work well) and now have some much better results. What kind of
bandwidth do other users see with TCP on qperf?

I will try with different PCI latencies set in the bios but are there
any other tweaks I should look at? 

Config:
Tyan S4989 Quad Opteron
Opteron 8350
16Gb Ram
Mellanox 20Gb PCIE-X8 card
Direct Connect to identical system
Centos 5.2 & OFED 1.3.1

Results:
[root at mrtickle device]# qperf 192.168.10.160 tcp_bw tcp_lat
tcp_bw:
    bw  =  671 MB/sec
tcp_lat:
    latency  =  31.1 us

[root at mrtickle device]# qperf 192.168.10.160 rc_bi_bw
rc_bi_bw:
    bw  =  1.71 GB/sec

-----Original Message-----
From: general-bounces at lists.openfabrics.org
[mailto:general-bounces at lists.openfabrics.org] On Behalf Of Robert
Dunkley
Sent: 24 July 2008 10:32
To: Or Gerlitz
Cc: general at lists.openfabrics.org
Subject: RE: [ofa-general] Direct / "Crossover" IPOIB

Thanks Or,

I have installed and run opensm on one of the machines, I can now move
traffic between them.

Pings show a latency og 0.07-0.09ms which seems OK.

Qperf shows abysmal results in connected mode though:
[root at mrhappy OFED-1.3.1]# qperf 192.168.10.161 tcp_bw tcp_lat
tcp_bw:
    bw  =  32.8 KB/sec
tcp_lat:
    latency  =  42.2 us

Datagram mode shows better bandwidth but still awful latency:
[root at mrtickle OFED-1.3.1]# qperf 192.168.10.160 tcp_bw tcp_lat
tcp_bw:
    bw  =  400 MB/sec
tcp_lat:
    latency  =  41.2 us

Is there something wrong with my setup? (QPerf seems to think so)

Thanks,

Rob

-----Original Message-----
From: Or Gerlitz [mailto:ogerlitz at voltaire.com] 
Sent: 24 July 2008 09:44
To: Robert Dunkley
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general] Direct / "Crossover" IPOIB

Robert Dunkley wrote:
>
> I've just setup two machines linked by a single Inifiniband cable. The

> IPOIB side of things looks OK, ifconfig shows the interface and each 
> machine can ping itself on its Infiniband IP. They cannot ping each 
> other though and ibstat looks wrong. What do I need to do to make this

> setup work?
>
> Port 1:
>
> State: Initializing
>
> Physical state: LinkUp
>
> Base lid: 0
>
> SM lid: 0
>
you have to run SM (Subnet Manager), if you issue

$ opensm

on one of the nodes, it will do that.

Or.


The SAQ Group
Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
SEMTEC Limited trading as SAQ is Registered in England & Wales
Company Number: 06481952

http://www.saqnet.co.uk AS29219
SAQ Group Delivers high quality, honestly priced communication and I.T.
services to UK Business.
DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit :
Backups : Managed Networks : Remote Support.

Find us in http://www.thebestof.co.uk/petersfield

_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general