[ofa-general] Processing multiple WR's in MTHCA (other drivers).

[Krishna Kumar2]

Hi,

If I call post_send with multiple WR's (as a list of 'next' linked WR's)
from IPoIB
to the driver, is it guaranteed that those WR's will go out on the wire in
the same
sequence ? Will the send sequence be : WR[0], WR[1], .... WR[n]; or this
cannot
be guaranteed by the driver after it issues the doorbell ?

The reason to ask is that when I modify the networking + ipoib code to send
multiple
skbs, I see huge number of retransmissions (original code: 250, modified
code: 5000
or more) as though skbs were sent out of order, and any amount of staring
at the
code is not helping.

The code to send multiple WR's is (hopefully readable):

/*
 * Normally start_index is 0 to indicate send from WR[0], but if one
post_send
* fails, start_index is set to the index of the first untried WR by
ipoib_send(), and
* calls post_send() again. I have also not seen any failures from provider
and a
* subsequent retry that causes this retransmission to happen.
*/
static inline int post_send(struct ipoib_dev_priv *priv, u32 qpn,
                            int start_index, int num_skbs,
                            struct ib_send_wr **bad_wr)
{
        int ret;
        struct ib_send_wr *last_wr, *next_wr;

        last_wr = &priv->tx_wr[start_index + num_skbs - 1];

        /* Set Completion Notification for last WR */
        last_wr->send_flags = IB_SEND_SIGNALED;

        /* Terminate the last WR */
        next_wr = last_wr->next;
        last_wr->next = NULL;

        /* Send all the WR's in one doorbell */
        ret = ib_post_send(priv->qp, &priv->tx_wr[start_index], bad_wr);

        /* Restore send_flags & WR chain */
        last_wr->send_flags = 0;
        last_wr->next = next_wr;

        return ret;
}

(I am also getting a completion with multiple WR's in each completion
handler, which
is good).

Thanks,

- KK