[ofa-general] Direct / "Crossover" IPOIB

[Yevgeny Kliteynik]

Robert,

Not directly answering your question, but just some general info:

Robert Dunkley wrote:
> Config:
> Tyan S4989 Quad Opteron
> Opteron 8350
> 16Gb Ram
> Mellanox 20Gb PCIE-X8 card
> Direct Connect to identical system
> Centos 5.2 & OFED 1.3.1
> 
> Results:
> [root at mrtickle device]# qperf 192.168.10.160 tcp_bw tcp_lat
> tcp_bw:
>     bw  =  671 MB/sec

Don't know what *exactly* is the max bw for tcp_bw (to me the
number looks ok), but note that the bottleneck here is IPoIB.
You have a quad-core CPU, so you can run four parallel tests,
and their combined bw would be similar to RDMA traffic bw,
which is 1.71 GB/s in your case.

> tcp_lat:
>     latency  =  31.1 us
> 
> [root at mrtickle device]# qperf 192.168.10.160 rc_bi_bw
> rc_bi_bw:
>     bw  =  1.71 GB/sec

This one looks fine (I guess you have PCI Express Gen2, right?).

-- Yevgeny

> -----Original Message-----
> From: general-bounces at lists.openfabrics.org
> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Robert
> Dunkley
> Sent: 24 July 2008 10:32
> To: Or Gerlitz
> Cc: general at lists.openfabrics.org
> Subject: RE: [ofa-general] Direct / "Crossover" IPOIB
> 
> Thanks Or,
> 
> I have installed and run opensm on one of the machines, I can now move
> traffic between them.
> 
> Pings show a latency og 0.07-0.09ms which seems OK.
> 
> Qperf shows abysmal results in connected mode though:
> [root at mrhappy OFED-1.3.1]# qperf 192.168.10.161 tcp_bw tcp_lat
> tcp_bw:
>     bw  =  32.8 KB/sec
> tcp_lat:
>     latency  =  42.2 us
> 
> Datagram mode shows better bandwidth but still awful latency:
> [root at mrtickle OFED-1.3.1]# qperf 192.168.10.160 tcp_bw tcp_lat
> tcp_bw:
>     bw  =  400 MB/sec
> tcp_lat:
>     latency  =  41.2 us
> 
> Is there something wrong with my setup? (QPerf seems to think so)
> 
> Thanks,
> 
> Rob
> 
> -----Original Message-----
> From: Or Gerlitz [mailto:ogerlitz at voltaire.com] 
> Sent: 24 July 2008 09:44
> To: Robert Dunkley
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] Direct / "Crossover" IPOIB
> 
> Robert Dunkley wrote:
>> I've just setup two machines linked by a single Inifiniband cable. The
> 
>> IPOIB side of things looks OK, ifconfig shows the interface and each 
>> machine can ping itself on its Infiniband IP. They cannot ping each 
>> other though and ibstat looks wrong. What do I need to do to make this
> 
>> setup work?
>>
>> Port 1:
>>
>> State: Initializing
>>
>> Physical state: LinkUp
>>
>> Base lid: 0
>>
>> SM lid: 0
>>
> you have to run SM (Subnet Manager), if you issue
> 
> $ opensm
> 
> on one of the nodes, it will do that.
> 
> Or.
> 
> 
> The SAQ Group
> Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
> SEMTEC Limited trading as SAQ is Registered in England & Wales
> Company Number: 06481952
> 
> http://www.saqnet.co.uk AS29219
> SAQ Group Delivers high quality, honestly priced communication and I.T.
> services to UK Business.
> DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit :
> Backups : Managed Networks : Remote Support.
> 
> Find us in http://www.thebestof.co.uk/petersfield
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>