[openib-general] Reregister Memory Region Verb

[Fab Tillier]

> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> Sent: Wednesday, January 25, 2006 10:49 AM
> 
> Fab Tillier wrote:
> >> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> >> Sent: Wednesday, January 25, 2006 10:06 AM
> >>
> >> Fab Tillier wrote:
> >>>> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> >>>> Sent: Wednesday, January 25, 2006 9:22 AM
> >>>>
> >>>>>> although we would prefer that it wouldn't block if possible
> >>>>>
> >>>>> mmm. All the current memory registration verbs both user and
> >>>>> kernel are blocking, is it an issue for you?
> >>>>>
> >>>>
> >>>> If you need to do memory registrations in a context where blocking
> >>>> is not an option then you really need FMR work requests as in RDMAC
> >>>> and InfiniBand 1.2 verbs.
> >>>
> >>> No.  The blocking semantics of memory registration APIs is a
> >>> deliberate design choice and not a limitation of the hardware.  It
> >>> is possible (though more complicated) to make the API asynchronous.
> >>> No existing IB stack to date has ever done so, however.
> >>>
> >>> - Fab
> >>
> >> If asynchronous memory registration (via work request) is such a bad
> >> idea then why is it part of both the RDMAC iWARP and InfiniBand 1.2
> >> verb specifications?
> >
> > You misunderstood me.  I didn't say anything about FMR being
> > a bad idea, just that regular MRs could be made to work in a
> > non-blocking manner.  Non-blocking calls don't require FMR, it could
> > be done without.
> 
> Yes, it is possible to specify a set of conditions where
> a memory registration verb would not have to block. But
> is it worthwhile to specify that under those conditions
> that it MUST NOT block?
> 
> For verification purposes it is much simpler if a given
> verb is either guaranteed to never block, or is considered
> subject to blocking. It is much easier to check whether a
> routine that is supposed to be non-blocking NEVER makes a
> call to a routine that could block than it is to check that
> it never makes a call to a routine with the set of conditions
> that might cause it to block.
> 
> So if applications have need to do registration where they
> are *guaranteed* that they will not block then I believe
> an asynch API (i.e., work requests) is a much better
> solution than adding lots of asterisks explaining
> when a call that normally "can block" will in fact
> be guaranteed not to block. The list of non-blocking
> scenarios is real easy to generate as a SHOULD NOT
> list, but it gets very tricky if you convert it to
> a MUST NOT list.

I wholeheartedly agree that having an API that *may* block is much worse than
just treating it as always blocking from a maintenance perspective.  What I was
referring to was that all the verb APIs could be made asynchronous, putting the
burden on the API provider to handle any blocking issues and not the end user.
You don't need work requests to have asynchronous APIs.  However, this is a
pretty significant change that I don't see happening for Linux (however I've had
a long term dream of making async verbs a reality in Windows for kernel
clients).

- Fab