[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Dror Goldenberg]

-----Original Message-----
From: Or Gerlitz [mailto:ogerlitz at voltaire.com] 
Sent: Wednesday, May 21, 2008 12:25 PM
To: Steve Wise
Cc: rdreier at cisco.com; general at lists.openfabrics.org; Dror Goldenberg
Subject: Re: [ofa-general] [PATCH RFC v3 1/2] RDMA/Core:
MEM_MGT_EXTENSIONS support

Steve Wise wrote:
>>> Support for the IB BMME and iWARP equivalent memory extensions ... 
>>> Usage Model:
>>> - MR allocated with ib_alloc_mr()
>>> - Page lists allocated via ib_alloc_fast_reg_page_list().
>>> - MR made VALID & bound to a specific page list via
>>> ib_post_send(IB_WR_FAST_REG_MR) - MR made INVALID via
>>> ib_post_send(IB_WR_INVALIDATE_MR)
>> AFAIK, the idea was to let the ulp post --two-- work requests, where 
>> the first creates the mapping and the second sends this mapping to 
>> the remote side, such that the second does not start before the first

>> completes (i.e a fence).
>>
>> Now, the above scheme means that the ulp knows the value of the 
>> rkey/stag at the time of posting these two work requests (since it 
>> has to encode it in the second one), so something has to be clarified

>> re the rkey/stag here, do they change each time this MR is used? how 
>> many bits can be changed, etc.
>
> The ULP knows the rkey/stag because its returned up front in the 
> ib_alloc_fast_reg_mr().  And it doesn't change (ignoring the key issue

> which we haven't exposed yet to the ULP).  The same rkey/stag can be 
> used for multiple mappings.  It can be made invalid at any point in 
> time via the IB_WR_INVALIDATE_MR so the fact that you're leaving the 
> same rkey/stag advertised is not a risk.
I understand that this (same rkey/stag used for all mapping produced for
a specific mr) is what you are proposing, I still think there's a chance
that by the spec and (not less important!) by existing HW support, its
possible to have a different rkey/stag per mapping done on an mr, for
example the IB spec uses a "consumer owned key portion of the L_Key" 
notation which makes me think there should be a way to have different
rkey per mapping, Roland? Dror?


[dg] When you post a fast register WQE, you specify the new 8 LSBits to
be assigned to the MR. The rest 24 MSBits are the ones that you obtained
while allocating the MR, and they persist throughout the lifetime of
this MR.