[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Christoph Lameter]

> On Dec 5, 2013, at 15:11, Jason Gunthorpe <jgunthorpe at obsidianresearch.com> wrote:
> 
>> On Thu, Dec 05, 2013 at 02:40:07PM -0600, Christoph Lameter wrote:
>> 
>> specific vendor library. A vendor could provide an inline function
>> f.e. to poll the cq by reading/writing memory mapped location
>> directly from within the app.
> 
> To do this a vendor just has to provide their own inlines for the hot
> path function call tables, it isn't technically very challenging..

Right. It would also allow the compiler to further optimize the code.

> Considering many environments are homogenous this seems like a
> worthwhile optimization possibility - maybe there could be some way to
> automate the construction of a hardware specific header..

We could provide a skeleton header that shows some examples on how to implement a basic inline function for most critical calls and then let them crib from there.

> At a minimum it probably suggests that the inline function table
> wrappers should be kept in a dedicated single-purpose header file.

The simplest approach would be to prescribe a set of functions that have to be provided by upper layers and then require each driver to implement these functions either in inline or explicit.

A simple app could directly link to the vendor libraries using the vendor headers. At the very low level only an API is prescribed without any need for intermediate code. Such an approach would allow to stay as close to hardware as possible and would result in the ability to work with minimal latency due to l1 cache misses.

An optional abstraction layer on top could then work with function vectors to multiplex multiple device drivers and could generate functions from the vendor inline code as required. Use of such an abstraction layer would then of course incur a penalty for the application that wants to be able to work using these abstractions with the devices for multiple vendors.

>> If we just want the indirection inline then that can also be done
>> with a library approach by setting up vectors there. dlopen and
>> friends can then be used to populate that table and the same
>> approach that ibverbs does today would still work.
> 
> Well, the flow is pretty much the same no matter what you do..
> - dlopen the provider libraries (or use linked-in libraries)
> - call the provider to get a function table related to the
>   ibv_device object
> - Use that table to call the provider to get a function table related
>   to the ibv_context object
> - .. recurse down for all sub ojects as per Sean's fabric proposal.
> 
> So dlsym doesn't really do anything terribly special for you. You
> still want the provider code to populate the function tables.
> 
> This is what libverbs does today.

Yes. I just wanted to point out that the VFIO approach could also do what we currently do. 

> This is also why it is so hard to split things up into more
> libraries. There needs to be only one provider library per HW.  There
> is private shared state encoded in the various objects it creates. So
> splitting the provider library into many (mlx_rdma, mlx_timestamp,
> mlx_foo) creates a mess where those libraries now need to
> cross-reference each others private structures.

I agree. I hate the current mess that we have to deal with.