[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

Roland Dreier wrote:
>  > Actually thinking about it, another possibility would be to wrap up the
> 
>  > newrkey = (mr->rkey & 0xffffff00) | newkey;
> 
>  > operation in a little inline helper function so people don't screw it
>  > up.  Maybe that's the cleanest way to do it.
> 
> If we add a "key" field to the work request, then it seems too easy for
> a consumer to forget to set it and end up passing uninitialized garbage.
> If the consumer has to explicitly update the key when posting the work
> request then that failure is avoided.
> 
> HOWEVER -- if we have the consumer update the key when posting the
> operation, then there is the problem of what happens when the consumer
> posts multiple fastreg work requests at once (ie fastreg, local inval,
> new fastreg, etc. in a pipelined way).  Does the low-level driver just
> take the the key value given when the WR is posted, even if there's a
> new value there by the time the WR is executed?
> 

I would have to say yes.  And it makes sense i think.

say rkey is 0x010203XX.  The a pipeline could look like:

fastreg (mr->rkey is 0x01020301)
rdma read (mr->rkey is 0x01020301)
invalidate local with fence (mr->rkey is 0x01020301)
fastreg (mr->rkey is 0x01020302)
rdma read (sink mr->rkey is 0x01020302)
invalidate local with fence (mr->rkey is 0x01020302)

So the consumer is using the correct mr->rkey at all times even though 
the rnic is possibly processing the previous generation (that was copied 
into a fastreg WR at an earlier point in time) at the same time as the 
app is registering the next generation of the rkey.

Steve.