[openib-general] ia64 perf and FMR

[Grant Grundler]

On Fri, Apr 01, 2005 at 06:40:48PM -0800, Grant Grundler wrote:
> Hi,
> Just wanted to share initial perf results (and surprise)
> that I'm getting on the HP ZX1/IA64 boxes.
> 
> Before FMR support was committed, netperf was reporting around
> 1720 Mb/s (215 MB/s) for IPoIB with msi_x=1 and netserver pinned
> to the CPU that wasn't taking interrupts. After FMR was committed,
> netperf is reporting about 3500 Mb/s (437 MB/s) for IPoIB. CPU was
> saturated on the send side in all cases.

FMR is a red herring.  I tried SVN r2080 and it has roughly the same
performance as r2082 (when FMR was committed) and later r210x.
"packed" attribute is a red herring too.

Performance stunk with r2050 and I will do a binary search this
week until I sort out which changes doubled the perf. ISTR
there was one change related  to a "double mapping" issue and I
will be tracking that down in a few days.

> I've a vague idea what "Fast Memory Registration" is but not a good
> understanding.  Can someone point me at a decent explanation of FMR?

I'm still fishing for this.
Even tips on which docs I might scrounge through are welcome.

...
> Maybe I'm just not configuring libsdp.conf correctly for netperf
> and maybe the IPoIB numbers are correct.

This was in fact the case.
The explanations aren't very good in the default .conf file.
Is there other documentation to describe libsdp.conf file?

"match program *" worked. Variations of "match destination"
and "match listen *:12866" did not. Well, it might have worked
for one side or the other, but not both.

I'm now getting ~5300-5500 Mb/s (~660 MB/s) using SDP with netperf.
(256KB socket size....probably too small).
So HP ZX1 chipset is doing quite well for a 3yr old PCI-X chipset.

thanks,
grant