[ofa-general] [PATCH RFC v4 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

>  enum ib_send_flags {
> @@ -676,6 +683,19 @@ struct ib_send_wr {
>  			u16	pkey_index; /* valid for GSI only */
>  			u8	port_num;   /* valid for DR SMPs on switch only */
>  		} ud;
> +		struct {
> +			u64				iova_start;
> +			struct ib_mr 			*mr;
> +			struct ib_fast_reg_page_list	*page_list;
> +			unsigned int			page_shift;
> +			unsigned int			page_list_len;
> +			unsigned int			first_byte_offset;
> +			u32				length;
> +			int				access_flags;
> +		} fast_reg;
> +		struct {
> +			struct ib_mr 	*mr;
> +		} local_inv;
>  	} wr;
>  };

Ok, while writing a test case for all this jazz, I see that passing the 
struct ib_mr pointer to both IB_WR_FAST_REGISTER_MR and 
IB_WR_INVALIDATE_MR is perhaps bad.  Consider a chain of WRs: 
INVALIDATE_MR linked to a FAST_REGISTER_MR and passed to the provider 
via a single ib_post_send() call.  You can't do that if you want to bump 
the key value between the invalidate and the fast_reg with the new key, 
which is probably what apps want to do. You are forced, under this 
proposed API, to post the two WRs separately and call 
ib_update_fast_reg_key() in between the ib_post_send() calls.

Perhaps we should just pass in a u32 rkey for both WRs instead of the mr 
pointer?  Then the code could put the old rkey in the invalidate WR, and 
the newly updated rkey in the fast_reg WR and chain the two together and 
do a single post.

I think this is the way to go: change the fast_reg and local_inv unions 
to take a u32 rkey instead of a struct ib_mr *mr.

Thoughts?

Steve.