[openib-general] Reserved L_Key API (was Re: DMA mapping on sparc64)

[Tillier, Fabian]

> From: Roland Dreier [mailto:roland at topspin.com]
> Sent: Tuesday, September 14, 2004 8:39 PM
> 
> Only the L_Key of this MR would be valid, and it would always have
> local write access (to match the semantics of reserved L_Key).  If the
> HCA supports reserved L_Key, it can just return the same L_Key for
> every consumer.  If need be it can take the PD into account.
> 
> It is required for the consumer to call ib_dereg_mr() on this MR when
> exiting, but this can be a NOP for HCAs that support reserved L_Key.

I think this makes a lot of sense and looks good.

> 
> I would argue that this entry point should replace reg_phys_mr as a
> mandatory low-level driver function; this will simplify the
> implementation of consumers that use the API.  Devices that can't even
> simulate reserved L_Key like Tavor (and I don't know of any such
> devices -- even on Topspin's embedded platforms I could implement this
> API) could just register a giant address range in a normal physical MR
> (and even use pci_set_dma_mask() to limit the size of the MR to 4 GB
> if they're really limited).

I think you still need reg_phys_mr (or some way to get an RKEY) for
kernel clients that do RDMA (SRP or kernel SDP, for example).  Something
like an RKEY with translation off but PD enforcement (I don't think you
want to get rid of PD enforcement for that kind of usage).  Given this,
I would suggest keeping similar semantics as memory registration.

Since we need an input PD anyway, I suggest having a call like:

struct ib_mr *ib_reg_dma_mr(struct ib_pd *pd,
			     int mr_access_flags );

Depending on the mr_access_flags, the returned MR could have a valid
RKEY with PD enforcement.  If only local access is needed, the LKEY
could be the reserved LKEY if the device supports it.  This would enable
both your desired usage as well as future usage by kernel clients that
perform RDMA.  With the above call, I think ib_reg_phys_mr can be
eliminated.

Thoughts?

- Fab