[openib-general] mthca FMR correctness (and memory windows)

[Fabian Tillier]

On 3/20/06, Dror Goldenberg <gdror at mellanox.co.il> wrote:
>
> > From: Talpey, Thomas [mailto:Thomas.Talpey at netapp.com]
> > Sent: Monday, March 20, 2006 10:20 PM
> >
> > At 02:58 PM 3/20/2006, Roland Dreier wrote:
> > >If you want to invalidate each FMR individually, then  there's not
> > >much
> > >point in using FMRs at all.  Just register and unregister memory
> > >regions.
>
> I'd say +/- here, because map() speeds up the registration (no
> need for cmdif), and unmpap() may slow down the deregistration.
>
> >
> > Really? The idea of FMR, I thought, was to preallocate the
> > TPT entry up front (alloc_fmr), and then populate it with the
> > lightweight fmr api (map_phys_fmr).

You pre-alloc the MPT entry, but not the MTT entries.  You then
populate the MTT by doing posted writes to the HCA memory (or host
memory for memfree HCAs).

> > If I use ib_reg_phys_mr(), then I incur all the preallocation
> > overhead each time. Sure the benefit of an FMR isn't merely
> > that the hard work is deferred, opening a vulnerability while
> > it's pending??

I don't know if allocating MTT entries is really expensive.  What
costs is the fact that you need to do command interface transactions
to write the MTT entries, while FMRs support posted writes.

That is, just like with alloc_fmr, you need to reserve and format an
MPT for regular memory registrations, which is a command interface
transaction.  For memory registration, one or more commands precede
this to write to the MTT. Thus, a memory registration is at a minimum
a 2 command interface transaction operation, potentially more
depending on the size of the registration.

Deregistration and freeing (not unmapping) an FMR should be
equivalent, I would think.

I'd spoken with Dror about changing the implementation of memory
registration to always use posted writes, and we'd come to the
conclusion that this would work, though doing so was not the intended
usage and thus not something that was garanteed to work going forward.
 One of Dror's main concerns was that a future change in firmware
could break this.

Such a change would allow memory registration to require only a single
command interface transaction (and thus only a single wait operation
while that command completes).  I'd think that was beneficial, but
haven't had a chance to poke around to quatify the gains.

I'd still be interested in seeing regular registration calls improved,
as it's clear that an application that is sensitive about its security
must either restrict itself to send/recv, buffer the data (data copy
overhead), or register/unregister for each I/O.

As to using FMRs to create virtually contiguous regions, the last data
I saw about this related to SRP (not on OpenIB), and resulted in a
gain of ~25% in throughput when using FMRs vs the "full frontal" DMA
MR.  So there is definitely something to be gained by creating
virutally contiguous regions, especially if you're doing a lot of RDMA
reads for which there's a fairly low limit to how many can be in
flight (4 comes to mind).

- Fab