[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Jason Gunthorpe]

On Thu, Dec 05, 2013 at 06:46:57PM +0000, Hefty, Sean wrote:
> > > VFIO could be used as a base API for getting access to the device. Then
> > we may
> > > need to have some additional syscalls that may be required to provide
> > > functionality that the existing VFIO interface does not provide.
> > 
> > I think the kernel API issue is orthogonal - there is nothing in the
> > 'libibverbs' API that is strongly tied to the kernel?
> > 
> > Someone could write a VFIO provider for verbs, for instance...
> 
> I don't know that the API is strongly tied with the kernel, but I'm
> not as sure about the implementation.  There are several calls that
> do not go to the provider -- ibv_get/ack_async_event,
> ibv_create_comp_channel, ibv_get/ack_cq_event, plus a couple of
> others.

Hmm, those all look like entry points so libibverbs could someday be
revised to call back into the provider for them. VFIO providers would
have to use an eventfd or similar for their completion channels.

The other issue is that libibverbs doesn't have any way for a provider
to hook device discovery, it assumes everything is in sysfs.

Something to think about with libfabric: ensure the providers can
hook everything. Keep the RDMA kernel API stuff separated from the
function call mux and API code.

I admit I also have some interesting future projects that could be
accomplished with a VFIO RDMA provider...

Jason