[ofiwg] provider specific definitions

[Hefty, Sean]

This email is in response to a PR which adds an EFA specific extension:

https://github.com/ofiwg/libfabric/pull/7054

My question to the group is, what policy should we use for exporting provider specific definitions to applications? 

Option 1 is a provider specific header file.  For example:

https://github.com/ofiwg/libfabric/blob/main/prov/usnic/src/fi_ext_usnic.h

Given the contents, this option makes sense for usnic.

However, there are cases (e.g. PR 7054), where the extensions are minimal.  Given that provider extensions have been rare, I propose using fi_ext.h.  This would make it easier on the apps, and provide better visibility in the definitions to avoid unnecessary overlap between providers (to help avoid application coding errors).

A third alternative is for very simple extensions to be defined in one of the main header files (e.g. fi_endpoint.h).  This option would work for PR 7054.

- Sean