[ofiwg] ofiwg item: completion optimizations

[Hefty, Sean]

This is a second set of items for the next ofiwg.  These were mentioned briefly 1 or 2 meetings ago, but to recap:

There is a provider driven need to relax support for completion flags (FI_SEND, FI_RECV, FI_READ, FI_WRITE, etc.).  IIRC, I *think* this comes from a limitation on the transmit side, where the completion data does not indicate the type of operation (e.g. send versus read versus write).  Being forced to set these flags results in providers needing to track the command using internal contexts.

There is no way to report optimal CQ attributes, like there is for fabric, domain, and endpoint attributes.  Applications need to know the optimal size of a CQ, along with the optimal number of endpoints that may be attached.

There is a provider driven optimization that wants to restrict the type of endpoints (or operations) associated with a CQ or counter.  This results from having a partial on-load/off-load model, depending on the type of operation being used.  When a CQ is created, there is no information about the type of operation completions that will be written to it.  As a result, a provider may be forced to use internal contexts to track requests, even in cases where they would not necessarily be needed.  As an example, the verbs provider can support RMA operations completely offloaded, but tagged message processing requires host processing.  When handling completions, there is no way for the provider to know whether completion data can be passed directly to the application or must be intercepted.  The result is that all completions must be intercepted, which impacts performance.

Unconnected endpoints may receive messages from remote addresses that have not been inserted into the corresponding address vector.  There needs to be a way to report the address to the application, so that it can be inserted.