[openib-general] MAD queuing model

[Roland Dreier]

    Fab> Can we limit a RMPP send to a single WQE outstanding at a
    Fab> time?  That is, all RMPP sends would require a single slot in
    Fab> the send queue of the QP.  Each segment of the RMPP is sent
    Fab> one at a time, with subsequent segments waiting for the
    Fab> previous one's WR to complete.  While this is not as
    Fab> efficient as posting many work requests, especially in the
    Fab> case where the send queue is nearly empty, it does provide
    Fab> for simple fairness.  I would expect that the QP would turn
    Fab> these things around fast enough that it should be suitable.

Sure, this is a possibility too.  However this now creates another
potential fairness problem: so many MADs are being posted that by the
time the RMPP layer can get a completion and have its next send
executed, a timeout has occurred.

I would prefer to have consumers queue up as many MADs as they have
ready to send as early as possible.  This maximize the possibilities
for adding a smart queue scheduler in the future.

 - R.