[openib-general] ia64 perf and FMR

[Grant Grundler]

On Mon, Apr 04, 2005 at 07:06:53AM -0700, Roland Dreier wrote:
> A memory region (MR) is a memory translation mapping in the HCA's
> context.  Usually, we create MRs via a firmware command, which is
> prohibitively expensive to do in the data path.  However, it is
> possible for the driver to write directly into the HCA's context,
> bypassing the firmware.  This is very cheap, just some posted writes,
> and so we can do it in the data path.  For example, for AIO, SDP uses
> this to map a bunch of random userspace pages into something virtually
> contiguous in the HCA's memory map, so that it can be used as for RDMA.

Thanks! I understood about 1/2 of that before. I'd like to read
a bit more detail though...

> However this shouldn't affect IPoIB in the least since a) it doesn't
> do any dynamic memory registration and b) it doesn't call any FMR
> functions anyway.

*nod* - FMR is clearly a red herring in this case.

> This is really odd.  I don't see how FMRs could directly change IPoIB
> performance, since IPoIB isn't using FMRs, even indirectly.

Sorry - I said "FMR" when I should have said r210x release. FMR
was just recently committed and I assumed (there's that word again)
that was related somehow. My bad.

> If SDP is
> not loaded, then I don't see how it could be used, but the fact that
> you get the same number for SDP and IPoIB really makes me think that
> the IPoIB number is really an SDP number.

Nope - those really where IPoIB numbers.

...
> The one that might make a difference is struct mthca_eqe in
> mthca_eq.c.  Unfortunately simply removing the packed attribute will
> break things on 64 bit archs unless the structure is written slightly
> differently.  It shouldn't be that difficult, so I should have
> something for you to test in a day or two.

Ok. I won't be able to test that until next week...I'll note which
rev picks that up and make sure to test it seperately.

BTW, SDP uses "packed" for a dozen or so structures. I haven't looked
at any q-syscollect or pfmon data yet to see where SDP is spending time
or if "packed" is part of the code path. But I don't have the impression
SDP is CPU bound like IPoIB is.

thanks,
grant