[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Jason Gunthorpe]

On Thu, Dec 05, 2013 at 04:19:46PM -0600, Christoph Lameter wrote:

> > To do this a vendor just has to provide their own inlines for the hot
> > path function call tables, it isn't technically very challenging..
> 
> Right. It would also allow the compiler to further optimize the code.

Yes, and linking to the provider directly and use a relocation instead
of a table lookup with provide some additional tiny savings.
 
> > Considering many environments are homogenous this seems like a
> > worthwhile optimization possibility - maybe there could be some way to
> > automate the construction of a hardware specific header..
> 
> We could provide a skeleton header that shows some examples on how
> to implement a basic inline function for most critical calls and
> then let them crib from there.

So, if we want to make this a goal I think it should be automatic, eg
you implement your provider code according to a certain pattern, run
script XYZ and you get the provider bypass header file.
 
> > At a minimum it probably suggests that the inline function table
> > wrappers should be kept in a dedicated single-purpose header file.
> 
> The simplest approach would be to prescribe a set of functions that
> have to be provided by upper layers and then require each driver to
> implement these functions either in inline or explicit.

So, here I how I see it could look..

The normal libfabric path (fabric.h implicit include):

 #ifdef LIBFABRIC_USE_MLX4_PROVIDER
 #include <fabric-mlx4-provider.h>
 #else
 static inline int indirect_func(OBJ *obj....) {return obj->function_table[..](obj,...);}
 #endif

Optimized full inline provider path (fabric-mlx4-provider.h):
 static inline int indirect_func(OBJ *obj....)
 {
    mlx4_private *priv = mlx4_get_private(obj);
    return readl(priv->base + REG) & ...;
 }

Optimized relocation based call-out (fabric-mlx4-provider.h):
 #define LIBFRABRIC_HAVE_PROVIDER_OVERRIDES 1
 static inline int indirect_func(OBJ *obj....) {return mlx4_unique_indirect_func(obj,...);}

App:

// gcc -DLIBFABRIC_USE_MLX4_PROVIDER=1 -lfabric -lfabric-mlx4
#include <fabric.h>

int res = indirect_func(obj,...);

To make it work the providers would have to have exported symbols with
globally unique names for all of their optimized function table entry
points.

This is obviously only worth doing for low level high speed functions,
and I think only worth doing if the construction of all this can be
automated somehow.

So it becomes a specific opt-in kind of optimization, and distros
obviously wouldn't use it. You'd have to compile your app locally with
the special gcc flags to get the benifit.

It is obviosly a micro optimization, only question is if it worth the
effort to develop it....

> > This is what libverbs does today.
> 
> Yes. I just wanted to point out that the VFIO approach could also do what we currently do. 

VFIO doesn't change the picture much, just some details of how the
provider library functions change..
 
> > This is also why it is so hard to split things up into more
> > libraries. There needs to be only one provider library per HW.  There
> > is private shared state encoded in the various objects it creates. So
> > splitting the provider library into many (mlx_rdma, mlx_timestamp,
> > mlx_foo) creates a mess where those libraries now need to
> > cross-reference each others private structures.
> 
> I agree. I hate the current mess that we have to deal with.

I would like to see libfabric just include the providers in source and
probably in binary. The provider library and ABI is just another
hassle to deal with that has ended up serving no purpose.

Especially since the providers are so small..

-- 
Jason Gunthorpe <jgunthorpe at obsidianresearch.com>        (780)4406067x832
Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada