[ofiwg] definition of a completion in OFI

[Jason Gunthorpe]

On Wed, Mar 04, 2015 at 11:27:10PM +0000, Hefty, Sean wrote:
> > Essentially what you are talking about here is adding a synchronous
> > 'end of stream mark' like TCP - the mark pushes through all data and
> > becomes visible as a guarenteed event on the far side.
> > 
> > Realistically, most apps that can close a connection need something
> > like this. I have fought with races in IB CM land in exactly this
> > area, and it is very difficult to resolve.
> 
> The intent with this option is to avoid adding implementation
> requirements on the completion processing.  E.g. if a provider
> generates a completion only after the request has been acked, then
> shutdown should be a no-op relative to completion processing.

What I read in your message is you have a connection oriented test,
but no support for end of stream in the library, so I wouldn't ever
expect that to work properly without races.

Completion processing itself really has nothing to do with clean
shutdown, and completions should only be expected to say the provider
is done with the resource.

Not supporting clean shutdown is fine, it just means apps have to
handle shutdown in-band, like IB already requires. Send an in-band
shutdown message and when the far side echo's an ACK then things can
be torn down. [And of course this is far more difficult with the new
message ordering modes]

In that case fi_close should probably be called fi_destroy to emphasize
the difference from sockets.

Jason