[Openframeworkwg] Some concerns about the new Fabric/IBverbs API

[Hefty, Sean]

> > > To do this a vendor just has to provide their own inlines for the hot
> > > path function call tables, it isn't technically very challenging..
> >
> > Right. It would also allow the compiler to further optimize the code.
> 
> Yes, and linking to the provider directly and use a relocation instead
> of a table lookup with provide some additional tiny savings.
> 
> > > Considering many environments are homogenous this seems like a
> > > worthwhile optimization possibility - maybe there could be some way to
> > > automate the construction of a hardware specific header..
> >
> > We could provide a skeleton header that shows some examples on how
> > to implement a basic inline function for most critical calls and
> > then let them crib from there.
> 
> So, if we want to make this a goal I think it should be automatic, eg
> you implement your provider code according to a certain pattern, run
> script XYZ and you get the provider bypass header file.
> 
> > > At a minimum it probably suggests that the inline function table
> > > wrappers should be kept in a dedicated single-purpose header file.
> >
> > The simplest approach would be to prescribe a set of functions that
> > have to be provided by upper layers and then require each driver to
> > implement these functions either in inline or explicit.
> 
> So, here I how I see it could look..
> 
> The normal libfabric path (fabric.h implicit include):
> 
>  #ifdef LIBFABRIC_USE_MLX4_PROVIDER
>  #include <fabric-mlx4-provider.h>
>  #else
>  static inline int indirect_func(OBJ *obj....) {return obj-
> >function_table[..](obj,...);}
>  #endif
> 
> Optimized full inline provider path (fabric-mlx4-provider.h):
>  static inline int indirect_func(OBJ *obj....)
>  {
>     mlx4_private *priv = mlx4_get_private(obj);
>     return readl(priv->base + REG) & ...;
>  }
> 
> Optimized relocation based call-out (fabric-mlx4-provider.h):
>  #define LIBFRABRIC_HAVE_PROVIDER_OVERRIDES 1
>  static inline int indirect_func(OBJ *obj....) {return
> mlx4_unique_indirect_func(obj,...);}
> 
> App:
> 
> // gcc -DLIBFABRIC_USE_MLX4_PROVIDER=1 -lfabric -lfabric-mlx4
> #include <fabric.h>
> 
> int res = indirect_func(obj,...);
> 
> To make it work the providers would have to have exported symbols with
> globally unique names for all of their optimized function table entry
> points.
> 
> This is obviously only worth doing for low level high speed functions,
> and I think only worth doing if the construction of all this can be
> automated somehow.
> 
> So it becomes a specific opt-in kind of optimization, and distros
> obviously wouldn't use it. You'd have to compile your app locally with
> the special gcc flags to get the benifit.
> 
> It is obviosly a micro optimization, only question is if it worth the
> effort to develop it....

If this is fairly straightforward and painless to do, then I think it's worth supporting.  PGAS may benefit from this level of optimization.

> 
> > > This is what libverbs does today.
> >
> > Yes. I just wanted to point out that the VFIO approach could also do what
> we currently do.
> 
> VFIO doesn't change the picture much, just some details of how the
> provider library functions change..
> 
> > > This is also why it is so hard to split things up into more
> > > libraries. There needs to be only one provider library per HW.  There
> > > is private shared state encoded in the various objects it creates. So
> > > splitting the provider library into many (mlx_rdma, mlx_timestamp,
> > > mlx_foo) creates a mess where those libraries now need to
> > > cross-reference each others private structures.
> >
> > I agree. I hate the current mess that we have to deal with.
> 
> I would like to see libfabric just include the providers in source and
> probably in binary. The provider library and ABI is just another
> hassle to deal with that has ended up serving no purpose.

The intent is for libfabric to include the provider source code, so there wouldn't be a need for, say, libmlx4.  In theory, a compiler should be able to inline the provider code into the application, but in practice, I haven't seen the compilers get past the function indirection, except in very simple cases.

- Sean