[ewg] IPoIB to Ethernet routing performance

[sebastien dugue]

  Hi,

  I know this might be off topic, but somebody may have already run into the same
problem before.

  I'm trying to use a server as a router between an IB fabric and an Ethernet network.

  The router is fitted with one ConnectX2 QDR HCA and one dual port Myricom 10G
Ethernet adapter.

  I did some bandwidth measurements using iperf with the following setup:

  +---------+               +---------+               +---------+
  |         |               |         |   10G Eth     |         |
  |         |    QDR IB     |         +---------------+         |
  | client  +---------------+  Router |   10G Eth     |  Server |
  |         |               |         +---------------+         |
  |         |               |         |               |         |
  +---------+               +---------+               +---------+

  
  However, the routing performance is far from what I would have expected.

  Here are some numbers:

  - 1 IPoIB stream between client and router: 20 Gbits/sec

    Looks OK.

  - 2 Ethernet streams between router and server: 19.5 Gbits/sec

    Looks OK.

  - routing 1 IPoIB stream to 1 Ethernet stream from client to server: 9.8 Gbits/sec

    We manage to saturate the Ethernet link, looks good so far.

  - routing 2 IPoIB streams to 2 Ethernet streams from client to server: 9.3 Gbits/sec

    Argh, even less that when routing a single stream. I would have expected
    a bit more than this.


  Has anybody ever tried to do some routing between an IB fabric and an Ethernet
network and achieved some sensible bandwidth figures?

  Are there some known limitations in what I try to achieve?


  Thanks,

  Sébastien.