[openib-general] ia64 perf and FMR

[Libor Michalek]

On Fri, Apr 01, 2005 at 06:40:48PM -0800, Grant Grundler wrote:
> Hi,
> Just wanted to share initial perf results (and surprise)
> that I'm getting on the HP ZX1/IA64 boxes.
> 
> Before FMR support was committed, netperf was reporting around
> 1720 Mb/s (215 MB/s) for IPoIB with msi_x=1 and netserver pinned
> to the CPU that wasn't taking interrupts. After FMR was committed,
> netperf is reporting about 3500 Mb/s (437 MB/s) for IPoIB. CPU was
> saturated on the send side in all cases.
> 
> I've a vague idea what "Fast Memory Registration" is but not a good
> understanding.  Can someone point me at a decent explanation of FMR?

  A few people responded with good FMR descriptions. However, I'd like
to add that when using an unmodified version of netperf, neither IPoIB
or SDP are using FMRs. IPoIB never uses FMRs, and currently SDP only
uses FMRs when the application is using Linux AIO to read or write data
on the socket. In that instance if the buffers are larger then a
threshold value they will be registered using FMRs and the contiguous
address is then shared with the remote half of the connetion which can
then RDMA to/from the buffer. The example code (ttcp.aio.c) I checked
in will use AIO and FMRs if the transfer size (-l) is over the 5K
default threshold.


-Libor