[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

Steve Wise wrote:
> 
> 
> Roland Dreier wrote:
>>  > Actually thinking about it, another possibility would be to wrap up 
>> the
>>
>>  > newrkey = (mr->rkey & 0xffffff00) | newkey;
>>
>>  > operation in a little inline helper function so people don't screw it
>>  > up.  Maybe that's the cleanest way to do it.
>>
>> If we add a "key" field to the work request, then it seems too easy for
>> a consumer to forget to set it and end up passing uninitialized garbage.
>> If the consumer has to explicitly update the key when posting the work
>> request then that failure is avoided.
>>
>> HOWEVER -- if we have the consumer update the key when posting the
>> operation, then there is the problem of what happens when the consumer
>> posts multiple fastreg work requests at once (ie fastreg, local inval,
>> new fastreg, etc. in a pipelined way).  Does the low-level driver just
>> take the the key value given when the WR is posted, even if there's a
>> new value there by the time the WR is executed?
>>
> 
> I would have to say yes.  And it makes sense i think.
> 
> say rkey is 0x010203XX.  The a pipeline could look like:
> 
> fastreg (mr->rkey is 0x01020301)
> rdma read (mr->rkey is 0x01020301)
> invalidate local with fence (mr->rkey is 0x01020301)
> fastreg (mr->rkey is 0x01020302)
> rdma read (sink mr->rkey is 0x01020302)
> invalidate local with fence (mr->rkey is 0x01020302)
> 
> So the consumer is using the correct mr->rkey at all times even though 
> the rnic is possibly processing the previous generation (that was copied 
> into a fastreg WR at an earlier point in time) at the same time as the 
> app is registering the next generation of the rkey.
> 

So something like this?

static void ib_update_fast_reg_key(struct ib_mr *mr, u8 newkey)
{
	/* iWARP: rkey == lkey */
	if (mr->rkey == mr->lkey)
		mr->lkey = mr->lkey & 0xffffff00 | newkey;
	mr->rkey = mr->rkey & 0xffffff00 | newkey;
}