[ofiwg] fork support and MR cache

[Hefty, Sean]

> > I think a reasonable hack could be to implement the behavior from my
> > first email.  That would involve bypassing the libibverbs calls and
> > going directly to the verbs providers for reg/dereg mr.
> 
> If you don't want the MADV then just don't call ibv_fork_init

The issue is that apps (OpenMPI) call ibv_fork_init directly.  The hack Brian is referring to is trying to detect if that has occurred, so the cache can be disabled.

> Please don't recode the kernel interface outside rdma-core, that stuff
> is hard to do right.

I'm was referring to calling:

mr->context->ops.reg_mr(...)

instead of indirectly through ibv_reg_mr().

That still goes through the rdma-core to reach the kernel.  In theory, we can use this to speed up registrations and enable a cache even if fork support is needed.

I'm actually fine disabling the cache if fork is enabled.  But how can that be detected?
 
- Sean