[openib-general] ip over ib throughtput

[Roland Dreier]

    Paul> Didn't see any response to Michael's numbers other than
    Paul> another mail today from Josh England saying 'great numbers
    Paul> till it dies'.

I've seen 225 MB/sec on my dual 2.2 GHz Xeon/PCI-X systems with NPtcp,
and on Josh's systems I've seen 425 MB/sec.  I'm pretty sure Josh's
problems will be solved by upgrading from FW 4.6.1 to 4.6.2.

    Paul> Are these results considered good on a 4x link? I realise
    Paul> there is a significant TCPIP overhead, but is this
    Paul> significantly better or worse than the vendor-specific
    Paul> equivalents using different drivers?

I believe the OpenIB IPoIB substantially outperforms any vendor stacks
(certainly I have never seen better numbers).  However IPoIB is a poor
use of IB, since it uses UD, which means every 2K packet requires its
own work request and completion to be generated and processed.

    Paul> What (short term?) plans are there for implementing a higher
    Paul> performance link that can show better transfer rates.

Nothing in the extremely short term but in the first half of 2005 I
would expect to see some preliminary implementations of the IPoIB
connected mode drafts, beta-level native MPI bindings and production
quality SDP.

    Paul> I've largely written off SDP going higher than ~300MB/s even
    Paul> with high CPU utilisation. (Assumptions: License concerns
    Paul> make it unlikely to be one of the first things openib
    Paul> tackle. Not trivial to implement it with zero copy or
    Paul> asynchronous I/O on Linux)

Libor can speak more authoritatively about his plans but I think the
main thing holding back SDP is completing the gen2 CM.  I believe
Libor already has zero copy and AIO working on kernel 2.6 on the gen1
stack, although there is some mthca work required ("FMRs") for this to
work on the gen2 stack.

    Paul> Am I right in thinking that the ib_verbs layer direct or
    Paul> ideally with MPI/uDAPL will be my best bet in the next 6
    Paul> months for showing a portable vendor-neutral implementation
    Paul> which might achieve 600MB/s transfers or slightly lower but
    Paul> with <25% cpu utilisation on PCIe.

I think in the next six months both MPI and SDP with AIO should be
running bus limited (ie close to 1 GB/sec) on PCIe with essentially
zero CPU usage.

 - Roland