[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

Or Gerlitz wrote:

> After discussing the rkey renew and fencing with send/rdma ops, I am 
> quite clear with how all this plugs well into ULPs such as SCSI or FS 
> low-level (interconnect) initiator/target drivers, specifically those 
> who use a transactional protocol. Few more points to clarify are (sorry 
> if it became somehow long):
> 
> * Do we want it to be a must for a consumer to invalidate an fast-reg mr 
> before reusing it? if yes, how?

The verbs specs mandate that the mr be in the invalid state when the 
fast-reg work request is processed.  So I think that means yes.  And the 
consumer invalidates it via the INVALIDATE_MR work request.

> 
> * If remote invalidation is supported, when the peer is done with the 
> mr, it sends the "response" in
> send-with-invalidate fashion and saves the mapper side from doing a 
> local invalidate. For the case of the mapping produced by SCSI initiator 
> or FS client, when remote invalidation is not supported, I don't see how 
> a local invalidate design can be made in a pipe-lined manner - Since 
> from the network perspective the I/O is done, the target response at 
> your hands, but until doing mr invalidation the pages are typically not 
> returned to the upper layer and the ULP has to "stall" till the 
> invalidation WR is completed? I don't say its a bug or a big issue, just 
> wonder what are your thoughts regarding this point.
> 

I guess that's why they invented send-with-inv, and read-with-inv-local.

> * talking about remote invalidation, I understand that it requires 
> support of both sides (and hence has to be negotiated), so the 
> IB_DEVICE_SEND_W_INV device capability says that a device can 
> send-with-invalidate, do we need a IB_DEVICE_RECV_W_INV cap as well?
> 
> * what about ZBVA, is it orthogonal to these calls, no enhancement of 
> the suggested API is needed even if zbva is used, or the other way, it 
> would work also when zbva is not used?
> 
> Or