[openib-general] makiing ibverb.h transport neutral -- 2nd draft

[Caitlin Bestler]

> 
> Work completions are used in speed path operations, so should 
> be optimized for caching, retrieving multiple completions, 
> etc., versus optimizing it for transport neutral code.  
> struct rdma_wc_storage isn't used anywhere in the API, so why 
> not make that a union of struct ib_wc and struct iwarp_wc?  
> (Of course this leads us back to unions if swapping the 
> naming convention...)
> 

Keep in mind that the most completions will be successful
SQ or recv completions, and these are transport neutral.

It is also something that is essential if you are trying
to write transport neutral middleware. Any given CQ will
be attached to a specific transport, but the middleware
would probably prefer not to think about that fact.

Of course they *could* think about it, by filling in 
either method A or method B in their higher level ops
vector based on whether they would call cq poll with
an iwarp_wc or ib_wc.

But I'm not seeing the benefit to the middleware in
terms of performance. My expectation is that they
will poll into a full size wc, then switch based
upon the information in the standard header. One chunk
won't deal with *any* non-successful completions (other
than to log the error), and others will deal with each
of the variants in a case statement (and/or function
that it invokes). After the split referencing the
specific sub-variant, by union or cast, has no
real performance penalty (just a typing nuisance penalty).

Having a union of all _wc types would be great, if there
was a way to get it without having to pick up all of 
the transport specific information. As noted in my
prior post, "open unions" aren't supported (nor are
"open enums" which would be great for accumulating
error codes).