[openib-general] MAD queuing model

[Fab Tillier]

> From: Roland Dreier [mailto:roland at topspin.com]
> Sent: Wednesday, September 01, 2004 1:43 PM
> 
>     Sean> I asked this more based on what the response to the first
>     Sean> question was, along with how the layering worked.  It also
>     Sean> comes down to some fairness, since while a large RMPP
>     Sean> request is being sent, responses for other MADs may be
>     Sean> queued behind it, which could result in timeouts on other
>     Sean> MADs.  Ideally, correct RMPP windowing would avoid this type
>     Sean> of condition.
> 
> I'd be inclined not to worry about this type of fairness for our
> initial implementation.  If it turns out that RMPP sends with 10
> megabyte windows are starving other sends, then we can add more
> sophisticated queue processing (since our MAD send queue handling will
> be nicely encapsulated in the core MAD layer ;).
> 
> For example one could have a scheduler that limits the number of
> consecutive sends with the same TID if other sends are waiting --
> however as I said I don't think this should be in our first version.
> 

Can we limit a RMPP send to a single WQE outstanding at a time?  That is,
all RMPP sends would require a single slot in the send queue of the QP.
Each segment of the RMPP is sent one at a time, with subsequent segments
waiting for the previous one's WR to complete.  While this is not as
efficient as posting many work requests, especially in the case where the
send queue is nearly empty, it does provide for simple fairness.  I would
expect that the QP would turn these things around fast enough that it should
be suitable.

Thoughts?

- Fab