[openib-general] posting send requests in RTR ( was Re: [PATCH 0/4] Dispatch communication related events to the IB CM)

[Michael S. Tsirkin]

Quoting r. Rimmer, Todd <trimmer at silverstorm.com>:
> The particular rare race is the case where:
> active side CMA consumer completes connection process (and CMA sends
> RTU).
> active side immediately sends a message
> passive side CQ callback occurs before CMA gets RTU or Communication
> Established Async event (and hence before CMA has moved QP to RTS)
> 
> While this race sounds rare, it's the kind of thing which will happen in
> some large cluster under heavy stress.  In which case it will be hard to
> debug, so its better to design out the race from the start.

IB spec is full of such subtle issues. How about disconnect MAD bypassing
the send? Protocol designers must be familiar with IB ordering rules.

> In this rare case, the passive side needs to queue any response TX it
> may want to do until it gets Communication established.

Not necessarily. CMA consumer at this point is a rare beast, but since the issue
is not specific to CMA, let's look at some IB protocols: IPoIB connected mode
can simply drop a packet.  So can SRP.  SDP (potential CMA consumer!) simply
never needs to send any application data before communication established event.

Possibly some ULPs may need an extra queue - so let them handle the complexity.

-- 
MST