[openib-general] Question about the IPoIB bandwidth performance ?

[Hal Rosenstock]

On Mon, 2006-06-05 at 11:12, hbchen wrote:
> Hi,
> I have a question about the IPoIB bandwidth performance.
> I did netperf testing using Single GiGE, Myrinet D card, Myrinet 10G
> ethernet card,
> and Voltaire Infiniband 4X HCA400Ex (PCI-Express interface).
> 
> 
> NIC (Jumbo enabled) Line bandwidth(LB) IPoverNIC bandwidth utilization
> (IPoNIC/LB)
> --------------------- ---------------- --------------
> ----------------------------------
> Single Gigabit NIC : 1Gb/sec=125MB/sec 120MB/sec 96% (PIC-X interface)
> Myrinet D card : 250MB/sec 240~-245MB/sec 96% ~ 98% (PCI-X interface)
> Myrinet 10G Ethernet: 10Gb/sec=1280MB/sec 980MB/sec 76.6% (My testing
> using Linux 2.6.14.6)
> (PCI-Express) 1225MB/sec 95.7% (Data from Myrinet website)
> IB HCA4X(PCI-Express): 10Gb/sec=1280MB/sec 420MB/sec 32.8% (My testing
> using Linux 2.6.14.6)
> 474MB/sec 37% (the best from OpenIB mailing list)
> (2.6.12-rc5 patch 1)
> 
> Why the bandwidth utilization of IPoIB is so low compared to the others
> NICs?

One thing to note is that the max utilization of 10G IB (4x) is 8G due
to the signalling being included in this rate (unlike ethernet whose
rate represents the data rate and does not include the signalling
overhead).

-- Hal

> There must be a lot of room to improve the IPoIB software to reach 75%+
> bandwidth utilization.
> 
> 
> HB Chen
> Los Alamos National Lab
> hbchen at labl.gov
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>