[libfabric-users] provider developer documentation

[Hefty, Sean]

Copying ofiwg.

> I'm looking for pointers to any documentation on writing a custom
> libfabric provider. I'm also looking for guidance as to the stability
> of the provider interfaces -- are custom providers supported from a
> compatibility standpoint, or would they generally break with a new
> libfabric version?

There's some documentation in the libfabric source tree docs/providers.  You can also look at prov/udp and prov/util as example code.  The rxd and rxm providers are 'utility' providers that add support for OFI features for providers that implement a minimal amount of the API.  RXD - RDM endpoint support over DGRAM endpoints.  RXM - RDM EPs over MSG EPs (requires that the MSG EP support RMA and shared receive contexts).  Depending on your provider, you may be able to make use of the rxd or rxm providers for faster bring-up.  Note that these providers are still in development.

The provider interfaces that are *exported* by libfabric for providers are considered stable.  This includes anything under include/rdma.  If they change, backwards compatibility will be maintained.  External providers are supported.

There are benefits of being in-tree, such as using the util code.  There are no guarantees that the utility code (lists, rbtree, util implementation of OFI constructs, etc.) used by providers will not change.  If you maintain a private provider outside of the upstream tree, you would need to resolve any changes to the util code if your provider makes use of it.  But that is a development issue that is separate from plugging into libfabric itself.  Utility code includes header files that are not installed -- see include/*.h.

- Sean