[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Jason Gunthorpe]

On Wed, Dec 04, 2013 at 06:39:07PM -0600, Christoph Lameter wrote:

> I am not clear on why we have this complicated API with the extension when the
> linker etc already provide such a framework with associated tools to manage
> them. There is a history for example of adding/modifying functions to glibc as
> well to the kernel. Do we really cannot use the established means of defining
> user space APIs?

The fundamental issue is that libibverbs serves as a mux between
multiple provider libraries and the application, while also needing
high performance.

In the general case there can be multiple providers loaded into the
same process. So you cannot use the dynamic linker to bind the app
to the provider.

With linker tricks the standard technique is to use an opaque
structure and some combination of symbol versioning and other tricks
to hide changes. However this requires every call through the ABI to
be a linker relocation.

For performance libibverbs inlines the linkage between the app and the
provider libarary via a function pointer which exposes all this mess
to everyone but the app developer..

So, to use the dynamic linker, you have to have two function calls per
high speed operation instead of one.

> Isnt the loader used to load an provider implementation that can then be used
> by ibverbs? Can these not be simple functions provided by an provider provided
> library instead of a vector of functions? dlopen and friends can manage that.

Yah, you can use dlopen to get the provider function pointers, but it
doesn't fundamentally help anything.
 
> If you must have these function pointers for certain use cases then the
> existing library mechanism can also provide that. Would not function
> definitions be better for the rest because we can then avoid the indirection?
> Plus we would have a rich set of tools to manage the APIs.

Right, everything that isn't a high speed op should be probably be a
relocated non-inline call using symbol versioning and opaque
structures.

> VFIO could be used as a base API for getting access to the device. Then we may
> need to have some additional syscalls that may be required to provide
> functionality that the existing VFIO interface does not provide.

I think the kernel API issue is orthogonal - there is nothing in the
'libibverbs' API that is strongly tied to the kernel?

Someone could write a VFIO provider for verbs, for instance...

Jasom