[openib-general] mthca FMR correctness (and memory windows)

[Talpey, Thomas]

At 12:25 PM 3/19/2006, Dror Goldenberg wrote:
>When ib_unmap_fmr() is done, you can be sure that the old FMR is 
>inaccessible. That's why this call blocks...

Okay, that's good.

>But Tom, I think that you should be looking at rdma/ib_fmr_pool.h for a
>better API
>to use for FMRs. This way you can allocate a pool and remap FMRs each 
>time you need one. You can look for examples in ulp/sdp directory.

Yeah, I noticed it but there is already a mechanism in the RPC/RDMA client
which supports memory windows, and it is easily adapted to include fmr's.
I might use the pool api later.

>> Is there a plan to make fmr's compliant with verbs 1.2?
>
>In the future... And it will probably be a different API, such an API
>that can go through a WQE->CQE.

Yes, it certainly will be different. I would prefer the CQ completion style
to the blocking style of the current fmr's. It would allow for better overlap
of RPC processing.

Currently I must defer the deregistration until processing reaches user
context, and then the blocking operation costs a context switch. With
the memory window API, I can launch the deregistration early, and it's
often polled as complete by the time I'm ready to return from the RPC. 
So, I would prefer that fmr's used a similar process.

>> Final question is memory windows. The code is there in the 
>> NFS/RDMA client, but all it gets is ENOSYS from ib_bind_mw(). 
>> (Yes, I know memory windows may not perform well on mthca. 
>> However, they're correct and hopefully faster than ib_reg_phys_mr()).
>
>FMR's the fastest. MWs are supported by the mthca HW. To my 
>knowledge there was no demand for MWs so far and that's why the
>code to handle them hasn't been implemented in mthca.

I want to quantify "fastest" before I agree with you. But I don't doubt
that they will perform better than memory windows, whose performance
on mthca hardware are disappointing (apparently) due to fencing and DMA
flushes. I could not achieve more than 150MB/sec using windows, while I
reached full bus bandwidth with a single "full-frontal" rkey. I am hoping that
fmr's come in closer to the latter.

I will not agree with your statement that nobody wants memory windows.
User space applications that don't wish to expose large amounts of memory
will certainly want them. Kernel space has the advantage here, by being
able to use fmr. User space can't do that.

Tom.