[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Hefty, Sean]

> That kind functionality is already provided by the shared library systems
> in Linux. Could not be each of the modules that we talked about just be
> mapped to a shared library and define the API simply that way? There are
> mechanisms in place to deal with compatibility issues.

These APIs are usable together, not stand-alone.  The calls used to report events are used with the data transfer calls.  And different data transfer calls work with the same endpoint (e.g. message send and RDMA write).
 
> What are the specific requirements that prevent us from using the standard
> library approach? If at all possible we should simply use what is there.

Can you provide more specific details on how you see this working?  I'm not following what sort of changes or approach you're thinking about.

This is what we have with libibverbs today:

libibverbs exports a set of function calls, such as

struct ibv_pd *ibv_alloc_pd(struct ibv_context *context);
struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, size_t length, int access);
struct ibv_cq *ibv_create_cq(struct ibv_context *context, int cqe,
     void *cq_context, ibv_comp_channel *channel, comp_vector);
struct ibv_qp *ibv_create_qp(struct ibv_pd *pd, ibv_qp_init_attr *qp_init_attr);
int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, attr_mask);
...

libibverbs itself can't implement these calls, since the implementation is dependent on the provider.  So the provider plugs a set of function calls into libibverbs.

struct ibv_context_ops {
	int			(*query_device)(struct ibv_context *context,
					      struct ibv_device_attr *device_attr);
	int			(*query_port)(struct ibv_context *context, uint8_t port_num,
					      struct ibv_port_attr *port_attr);
	struct ibv_pd *		(*alloc_pd)(struct ibv_context *context);
	int			(*dealloc_pd)(struct ibv_pd *pd);
	struct ibv_mr *		(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
					  int access);
	...

libibverbs can then map calls to the provider, in some cases using static inline function.  Whether a static inline call is used or not, the basic operation is the same.

static inline int ibv_poll_cq(struct ibv_cq *cq, int num_entries, struct ibv_wc *wc)
{
	return cq->context->ops.poll_cq(cq, num_entries, wc);
}

>From one viewpoint, I'm suggesting replacing 'struct ibv_context_ops' with 'struct ibv_qp_ops' + 'struct ibv_cq_ops' + 'struct ibv_mr_ops' + ..., and using static inline functions similar to ibv_poll_cq().

I don't see how we can get away from using function pointers into the providers.

> Then regarding the separation between user space and kernel space: There
> are alternate approaches possible to the use of IBverbs. The Linux kernel
> has added a VFIO interface that allows exposure of device driver controls
> to user space. If we could base libfabrics on that API instead then the
> RDMA approach would be generically be usable with any device that supports
> VFIO.
> 
> http://lwn.net/Articles/509153/

Are you suggesting using VFIO as the interface between user space and the kernel, rather than dropping through the kernel ib_user_verbs module?

Do you see the use of VFIO impacting the user space APIs or the internal architecture of the framework or both?

- Sean