[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Jason Gunthorpe]

On Thu, Dec 05, 2013 at 02:40:07PM -0600, Christoph Lameter wrote:

> specific vendor library. A vendor could provide an inline function
> f.e. to poll the cq by reading/writing memory mapped location
> directly from within the app.

To do this a vendor just has to provide their own inlines for the hot
path function call tables, it isn't technically very challenging..

Considering many environments are homogenous this seems like a
worthwhile optimization possibility - maybe there could be some way to
automate the construction of a hardware specific header..

At a minimum it probably suggests that the inline function table
wrappers should be kept in a dedicated single-purpose header file.

> If we just want the indirection inline then that can also be done
> with a library approach by setting up vectors there. dlopen and
> friends can then be used to populate that table and the same
> approach that ibverbs does today would still work.

Well, the flow is pretty much the same no matter what you do..
 - dlopen the provider libraries (or use linked-in libraries)
 - call the provider to get a function table related to the
   ibv_device object
 - Use that table to call the provider to get a function table related
   to the ibv_context object
 - .. recurse down for all sub ojects as per Sean's fabric proposal.

So dlsym doesn't really do anything terribly special for you. You
still want the provider code to populate the function tables.

This is what libverbs does today.

This is also why it is so hard to split things up into more
libraries. There needs to be only one provider library per HW.  There
is private shared state encoded in the various objects it creates. So
splitting the provider library into many (mlx_rdma, mlx_timestamp,
mlx_foo) creates a mess where those libraries now need to
cross-reference each others private structures.

If you can't split the provider then it doesn't make any sense to
split libfabric either, since at this level it is really just a
function call mux and ABI contract.

> How much of a benefit is the current inline indirection implementation in
> ibverbs.h for ib_post_send etc?

The other way to look at is is what use is adding a relocated function
call?  What does that help in future?

Remember, the ABI of every function call in the table is fixed as soon
as the call is added. It is fixed both by the user app calling it, and
the various provider libraries implementing it.

It is infeasible to change the signature for multiple reasons.

Jason