[ewg] IPoIB to Ethernet routing performance

[Richard Croucher]

Unfortunately, the 4036E only has two 10G Ethernet ports which will ultimately limit the throughput.  

The Mellanox BridgeX looks a better hardware solution with 12x 10Ge ports but when I tested this they could only provide vNIC functionality and would not commit to adding IPoIB gateway on their roadmap.  

Qlogic also offer the 12400 Gateway.  This has 6x 10ge ports.   However, like the Mellanox, I understand they only provide host vNIC support.

I'll leave it to representatives from Voltaire, Mellanox and Qlogic to update us. Particularly on support for InfiniBand to Ethernet Gateway for RoCEE.  This is needed so that RDMA sessions can be run between InfiniBand and RoCEE connected hosts.  I don't believe this will work over any of the today's available products.

Richard

-----Original Message-----
From: sebastien dugue [mailto:sebastien.dugue at bull.net] 
Sent: 06 December 2010 11:40
To: Richard Croucher
Cc: 'OF EWG'; 'linux-rdma'
Subject: Re: [ewg] IPoIB to Ethernet routing performance

On Mon, 6 Dec 2010 10:49:58 -0000
"Richard Croucher" <richard at informatix-sol.com> wrote:

> You may be able to improve by doing some OS tuning.

  Right, I tried a few things concerning the TCP/IP stack tuning but nothing
really came out of it.

>  All this data should stay in kernel mode but there are lots of bottlenecks in
> the TCP/IP stack that limit scalability.

  That may be my problem in fact.

>  The IPoIB code has not been optimized for this use case.

  I don't think IPoIB to be the bottleneck. In this case as I managed to feed
2 IPoIB streams between the client and the router yielding about 40 Gbits/s bandwidth.

> 
> You don't mention what Server, kernel and OFED distro you are running.

  Right, sorry. The router is one of our 4 sockets Nehalem-EX box with 2 IOHs which
is running an OFED 1.5.2.

> 
> The best performance is achieved using InfiniBand/Ethernet hardware gateways.
> Most of these provide virtual Ethernet NICs to InfiniBand hosts, but the Voltaire
> 4036E does provide a  IPoIB to Ethernet gateway capability.  This is FPGA based
> so does provide much higher performance than you will achieve using a standard server solution.

  That may be a solution indeed. Are there any real world figures out there
concerning the 4036E performance?

  Thanks Richard,

  Sébastien.


> 
> -----Original Message-----
> From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of sebastien dugue
> Sent: 06 December 2010 10:25
> To: OF EWG
> Cc: linux-rdma
> Subject: [ewg] IPoIB to Ethernet routing performance
> 
> 
>   Hi,
> 
>   I know this might be off topic, but somebody may have already run into the same
> problem before.
> 
>   I'm trying to use a server as a router between an IB fabric and an Ethernet network.
> 
>   The router is fitted with one ConnectX2 QDR HCA and one dual port Myricom 10G
> Ethernet adapter.
> 
>   I did some bandwidth measurements using iperf with the following setup:
> 
>   +---------+               +---------+               +---------+
>   |         |               |         |   10G Eth     |         |
>   |         |    QDR IB     |         +---------------+         |
>   | client  +---------------+  Router |   10G Eth     |  Server |
>   |         |               |         +---------------+         |
>   |         |               |         |               |         |
>   +---------+               +---------+               +---------+
> 
>   
>   However, the routing performance is far from what I would have expected.
> 
>   Here are some numbers:
> 
>   - 1 IPoIB stream between client and router: 20 Gbits/sec
> 
>     Looks OK.
> 
>   - 2 Ethernet streams between router and server: 19.5 Gbits/sec
> 
>     Looks OK.
> 
>   - routing 1 IPoIB stream to 1 Ethernet stream from client to server: 9.8 Gbits/sec
> 
>     We manage to saturate the Ethernet link, looks good so far.
> 
>   - routing 2 IPoIB streams to 2 Ethernet streams from client to server: 9.3 Gbits/sec
> 
>     Argh, even less that when routing a single stream. I would have expected
>     a bit more than this.
> 
> 
>   Has anybody ever tried to do some routing between an IB fabric and an Ethernet
> network and achieved some sensible bandwidth figures?
> 
>   Are there some known limitations in what I try to achieve?
> 
> 
>   Thanks,
> 
>   Sébastien.
> 
> 
> 
> 
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> 
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg