[openib-general] mthca FMR correctness (and memory windows)

[Dror Goldenberg]

> From:  Talpey, Thomas
> Sent: Sunday, March 19, 2006 5:36 PM
> 
> I'm implementing FMR memory registration mode in the NFS/RDMA 
> client, and I've got it mostly working. However as I 
> understand it, mthca's existing fmr's do not guarantee that 
> the r_key is completely invalidated when the ib_unmap_fmr() 
> returns. This makes using them rather problematic, to say the least.

When ib_unmap_fmr() is done, you can be sure that the old FMR is 
inaccessible. That's why this call blocks...

> 
> Now, I notice that ib_unmap_fmr() is a blocking operation (at 
> least, the kernel whines about semaphores being waited in 
> interrupt context, when I experimented with that).
> 
> Does this mean mthca's ib_unmap_fmr() is waiting for the 
> invalidation now, or plans to in the future?

It is indeed waiting for true invalidation.


But Tom, I think that you should be looking at rdma/ib_fmr_pool.h for a
better API
to use for FMRs. This way you can allocate a pool and remap FMRs each 
time you need one. You can look for examples in ulp/sdp directory.

> 
> Is there a plan to make fmr's compliant with verbs 1.2?

In the future... And it will probably be a different API, such an API
that can go through a WQE->CQE.

> 
> Final question is memory windows. The code is there in the 
> NFS/RDMA client, but all it gets is ENOSYS from ib_bind_mw(). 
> (Yes, I know memory windows may not perform well on mthca. 
> However, they're correct and hopefully faster than ib_reg_phys_mr()).

FMR's the fastest. MWs are supported by the mthca HW. To my 
knowledge there was no demand for MWs so far and that's why the
code to handle them hasn't been implemented in mthca.

> 
> What is the plan to implement mthca memory windows?
> 
> Thanks,
> Tom.