[ewg] IPoIB to Ethernet routing performance

[Richard Croucher]

IPoIB is far easier to use and does not carry out the additional management
burden of vNICS.

With vNICs you have to manage the MAC address mapping to Ethernet g/w port.
In some situations, such as when multiple G/w's are used for resiliency this
can amount to a lot of separate vNICs on each server to manage.  In a small
configuration I had, we ended up with 6 vNICS per server to manage.  On a
large configuration this additional management would be a big burden.

My experience with IPoIB has always been very positive. All my existing
socket programs have worked, even some esoteric ioctls I use for multicast
and buffer  management.
Performance could always be better, but in my experience it's not great for
the vNICS either.   Latency in particular was very disappointing when I
tested.  
If you want high performance you have to avoid TCP/IP.

-----Original Message-----
From: Jabe [mailto:jabe.chapman at shiftmail.org] 
Sent: 27 December 2010 11:51
To: richard.croucher at informatix-sol.com
Cc: Richard Croucher; 'Ali Ayoub'; 'Christoph Lameter'; 'linux-rdma';
'sebastien dugue'; 'OF EWG'
Subject: Re: [ewg] IPoIB to Ethernet routing performance

On 12/26/2010 11:57 AM, Richard Croucher wrote:
> The vNIC driver only works when you have Ethernet/InfiniBand hardware
> gateways in your environment.   It is useful when you have external hosts
to
> communicate with which do not have direct InfiniBand connectivity.
> IPoIB is still heavily used in these environments to provide TCP/IP
> connectivity within the InfiniBand fabric.
> The primary Use Case for vNICs is probably for virtualization servers, so
> that individual Guests can be presented with a virtual Ethernet NIC and do
> not lead to load any InfiniBand drivers.  Only the hypervisor needs to
have
> the InfiniBand software stack loaded.
> I've also applied vNICs in the Financial Services arena, for connectivity
to
> external TCP/IP services but there the IPoIB gateway function is arguably
> more useful.
>
> The whole vNIC arena is complicated by different, incompatible
> implementations from each of Qlogic and Mellanox.
>
> Richard
>    


Richard, with your explanation I understand why vNIC / EoIB is used in 
the case you cite, but I don't understand why it is NOT used in the 
other cases (like Ali says).

I can *guess* it's probably because with a virtual ethernet fabric you 
have to do all IP stack in software, probably without even having the 
stateless offloads (so it would be a performance reason). Is that the 
reason?

Thank you