[ewg] IPoIB to Ethernet routing performance

[Jabe]

>    The router is fitted with one ConnectX2 QDR HCA and one dual port Myricom 10G
> Ethernet adapter.
>
> ...
>
>    Here are some numbers:
>
>    - 1 IPoIB stream between client and router: 20 Gbits/sec
>
>      Looks OK.
>
>    - 2 Ethernet streams between router and server: 19.5 Gbits/sec
>
>      Looks OK.
>    


Actually I am amazed you can get such a speed with IPoIB. Trying with 
NPtcp on my DDR infiniband I can only obtain about 4.6Gbit/sec at the 
best packet size (that is 1/4 of the infiniband bandwidth) with this 
chip embedded in the mainboard: InfiniBand: Mellanox Technologies 
MT25204 [InfiniHost III Lx HCA]; and dual E5430 xeon (not nehalem).
That's with 2.6.37 kernel and vanilla ib_ipoib module. What's wrong with 
my setup?
I always assumed that such a slow speed was due to the lack of 
offloading capabilities you get with ethernet cards, but maybe I was 
wrong...?
Also what application did you use for the benchmark?
Thank you