[openib-general] FMR and how they work

[Caitlin Bestler]

On 5/3/05, Libor Michalek <libor at topspin.com> wrote:
> On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote:
> > What advantages is this style of fast memory register supposed to
> > have over the work request style found in IB 1.2 and iWARP?
> 
>   It's faster since all you're doing is rewriting the device's
> page table mapping. It would be interesting to see benchmark
> results if anyone has them.
> 

Which is exactly what a Fast Memory Register work request does.
I also don't see the advantage versus doing a physical rereg.

There is an important difference with FMR Work Requests. They
 follow the normal ordering and completion rules and hence can
be pipelined and interleaved with send requests, just like memory
window binds.

If you only do Fast Memory Registration once, doing so directly from
the verbs would be ever so slightly faster because you would be
copying from the source to the PBL entries directly. While with
a work request you have to copy them (or a pointer to them) into
the work request first.

But if you are doing multiple FMRs, which would seem to be
implied if there is an actual need for them to be "fast", then
the ability to pipeline them is far more valuable.

More importantly, the FMR work request integrates with kDAPL
far better than a some special verb.
> > How is this style of fast memory register supposed to be utilized
> > from DAPL or ITAPI when they both assume work requests?
> 
>   It has yet to come up, so far all the consumers are written
> directly to IB kernel verbs.
> 
> -Libor
> 
As pointed out, at least some major consumers are written to
kDAPL. Additionally, is it appropriate for the Linux kernel to
endorse non-standard functionality in preference to standard
functionality? By default, shouldn't the IBTA's endorsement
of a FMR work request be taken as an industry consensus?
Especially since the other RDMA verbs (RMDAC and RNIC-PI)
also take this approach?