[ofiwg] FI_MSG_PREFIX & versioning

[Hefty, Sean]

I spent some time looking at the FI_MSG_PREFIX issue.  I think the best option is to introduce a new FI_MSG_PREFIX2 mode bit, with the updated definition.  The existing FI_MSG_PREFIX bit will be documented as deprecated, but otherwise left alone.

Trying to redefine the behavior of FI_MSG_PREFIX would break compatibility.

---

For versioning, I think it would be easier for users if the API and ABI versions were in sync.  This means bumping FI_MINOR_VERSION to match the ABI version (1.1).

In order to keep the major version the same, we should ensure that: binary applications continue to work, and an app can be recompiled without changes.  Anything that breaks this should bump the major version number.

Personally, I would rather see libfabric have at least 1-2 years of air time before considering a 2.0 release.

---

Changing fi_info (or other structures), is possible, but requires significantly more work than listed in pull request 1122.  Because of how the interfaces evolved, fi_getinfo() is not guaranteed to be the first API invoked by an application.

To change fi_info, we would need to do something similar to this:

1. Redefine the current fi_info to fi_info_1_0
2. Redefine fi_getinfo & other calls to fi_getinfo_1_0
3. The 1_0 calls should operate on struct fi_info_1_0 
4. Introduce new fi_info structure
5. Introduce new fi_getinfo & other calls
6. Use symver support to direct apps to the correct API
7. Perform several other updates that I'm too lazy to type, such as fixing fi_tostr

We need to use symver to know the structure format for calls such as fi_dupinfo/fi_allocinfo.

The version passed into fi_getinfo() indicates which fields the application knows to set, which is separate from the structure format version.

Adding a version field (or any new field) to fi_info would require these modifications. 

- Sean