[ofa-general] Re: IB/core: Add creation flags to QPs

[Or Gerlitz]

Jack Morgenstein wrote:
> Actually, we did not break the ABI for XRC (we had to do cartwheels to avoid this, but what the heck).
>
> The create_flags field was added ONLY to struct ib_qp_init_attr (kernel only),
> and was not exported upwards to the userspace API (so as not to break the ABI).
>
> I got around the create_flags problem by adding a new verb to userspace
> (ibv_create_xrc_rcv_qp() ) with its own ABI to kernel space.  Since the kernel space function (added to uverbs_cmd: ib_uverbs_create_xrc_rcv_qp() ) "knew" that it was creating an XRC_RCV qp, it set the flag in ib_qp_init_attr appropriately.
>
> Additionally, Eli did not need user-space involvement for his LSO flag -- so we escaped needing to increment the ABI version number.
>
> If we need to use the create_flags field in userspace, we WILL need to break the ABI.
>
> You can avoid doing this in one of 2 ways:
> Either:  create a new QP type
> Or:      add new verbs to the core.
Jack,

I am trying to figure out what would be the least painful way to allow 
for specifi

Looking on the libibverbs - XRC instance 
(git://git.openfabrics.org/ofed_1_3/libibverbs.git) and re-reading your 
email, my understanding is that the way you went was

1. add a new field at the end of struct ibv_qp_init_attr
2. add bunch of new XRC verbs packed in struct ibv_xrc_ops
3. both create_qp & create_xrc_rcv_qp get ibv_qp_init_attr, the latter 
looks on the xrc field
4. add ibv_xrc_ops at the end of the ibv_context

I have pasted below the relevant code from verbs.h, as I can't point on 
one patch that does this all, since there were some fixes/changes since 
the initial commit.

So if we want to have qp creation verb that gets creation flags, we can can

A. add create_flags field to the end of ibv_qp_init_attr
B. introduce  struct ibv_qp * (*create_qp_ext)(struct ibv_pd *pd, struct 
ibv_qp_init_attr *attr)
C. enhance struct ibv_context similarily to what was done for xrc 

This seems to bring to minimum the breakage from the perspective of 
libibverbs consumers.

As for taking this down to uverbs, I am fine with anything you suggest.

Or.
> struct ibv_qp_init_attr {
> 	void		       *qp_context;
> 	struct ibv_cq	       *send_cq;
> 	struct ibv_cq	       *recv_cq;
> 	struct ibv_srq	       *srq;
> 	struct ibv_qp_cap	cap;
> 	enum ibv_qp_type	qp_type;
> 	int			sq_sig_all;
> 	struct ibv_xrc_domain  *xrc_domain;
> };
>
> struct ibv_qp * (*create_qp)(struct ibv_pd *pd, struct ibv_qp_init_attr *attr);
>
> struct ibv_xrc_ops {
> 	...
> 	int (*create_xrc_rcv_qp)(struct ibv_qp_init_attr *init_attr, uint32_t *xrc_qp_num);
> 	...
> };
>
> struct ibv_context {
> 	struct ibv_device      *device;
> 	struct ibv_context_ops	ops;
> 	int			cmd_fd;
> 	int			async_fd;
> 	int			num_comp_vectors;
> 	pthread_mutex_t		mutex;
> 	void		       *abi_compat;
> 	struct ibv_xrc_ops     *xrc_ops;
> };