[ofiwg] definition of a completion in OFI

[Hefty, Sean]

> > However, an app using an unconnected endpoint should have some
> > reasonable expectation of things working.  Shutdown isn't usable
> > here.
> 
> It makes sense to me that unconnected end points should try very hard
> to put a packet on the wire after generating a completion. Ie in your
> sockets land you can still early complete and copy to a hidden buffer,
> but the buffer and socket must linger after close until packets hit
> the wire. This is essentially what TCP sockets do.

I agree, which is why I brought this up. 

> > > Completion processing itself really has nothing to do with clean
> > > shutdown, and completions should only be expected to say the
> > > provider is done with the resource.
> >
> > Then would you agree with applying my previous definition (slightly
> > modified) for fi_shutdown to fi_close?
> >
> > fi_close() - For reliable endpoints, blocks until all requests have
> > been responded to by their respective destination(s), or the
> > requests have timed out.  For unreliable endpoints, indicates that
> > all requests have successfully been transmitted into the fabric.
> 
> Really, if there is no such thing as an 'end of stream mark' then
> there is little sense in trying to talk about close or shutdown - you
> can't create anything resembling those conditions from sockets.
> 
> I think what you have is a unilateral destruction of the socket:
>  - UD cases try very hard to put completed sends on the wire
>  - Stop generating new recv completions, and retire all
>    unused WQEs
>  - No gurantees of data transfer and no synchronous end of stream for
>    RC.
> 
> For RC the other side perceives this as an asynchronous notification
> that the QP is destroy'd - either beacuse it got the CM mad, or
> because it got an error for a data packet. This is not an end of
> stream mark because it is not synchronous with the data transfer.

We can't be guaranteed of a kernel agent to do a linger.  Nor can we be guaranteed of IB or MADs .  An end of stream marker concept doesn't easily work for reliable unconnected.  I care about defining the expected behavior of the APIs, not the implementation. 

Whatever is defined for reliable unconnected case seems that it should work for UD or RC.  The app needs some level of assurance that after it calls fi_blah or receives an FI_BLAH event that all data transfers have completed and it's safe to free all resources.

> > > Not supporting clean shutdown is fine, it just means apps have to
> > > handle shutdown in-band, like IB already requires. Send an in-band
> > > shutdown message and when the far side echo's an ACK then things can
> > > be torn down. [And of course this is far more difficult with the new
> > > message ordering modes]
> >
> > In-band shutdown seems like it would suffer from the same problem.
> > The sender of the ACK posts the send, then tears things down before
> > the ACK is sent.
> 
> The tear down should be done using CM MADs, which provide the last bit
> of synchronization to cover off that case.
> 
> But the inband sequence is required first so the CM MAD's don't race
> with the QP generated packets.
> 
> This is very tricky and should be hidden and automatic for people who
> need it..

I agree, but the developer at least needs to know what calls to invoke and events to look for.  They need to know if a call like fi_close() is going to block for a minute while CM MADs are retried, or if they should call fi_shutdown() and wait for an FI_SHUTDOWN event. The apps needs to know when it's safe to call either function.  The app needs to know if it can receive 2 events for the same request...