[openib-general] QP sq->tail always = 0

[Roland Dreier]

    Jeff> For the server, it first runs: ib_post_recv() for N times
    Jeff> (N=5 here)

    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (1 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (2 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (3 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (4 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)

This looks like it has actually already posted 5 receives and is
posting 5 sends one by one.  Did you mean ib_post_send() above?

    Jeff> Then, the client sends N=5 responses via ib_post_recv() that
    Jeff> the server sees via ib_poll_cq().

Again, did you mean ib_post_send()?

    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 1 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 2 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 3 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 4 tail, 5 max)

    Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 5 tail, 5 max
    Jeff> ib_mthca 0000:09:00.0: SQ 050406 full (5 head, 0 tail, 5
    Jeff> max, 0 nreq)

    Jeff> At this point, the RQ is empty: (head == tail) And at this
    Jeff> point the SQ is full: ((head - tail) == max)

    Jeff> Is this a correct understanding?

It looks that way -- you seem to have posted 5 sends and 5 receives,
and you are getting 5 receive completions, which empties the receive
queue.

Are you seeing any send completions on the CQ attached to the send
queue?  One possibility is that you're not generating any send
completions because you created your CQ with a sq_sig_type of
IB_SIGNAL_REQ_WR and then posting send requests without the
IB_SEND_SIGNALED flag.

(The IB_SIGNAL_REQ_WR flag means that not all send requests posted to
the send queue will generate a completion -- only those marked with
the IB_SEND_SIGNALED flag.  However, the driver can't free a send
request from the send queue until it knows it has completed, and the
only way for the driver to know that is to see a completion for the
given request or a later request.  Requests on a queue always complete
in order, so if a later request completes and generates a completion,
the driver can also free any earlier unsignaled requests)

 - R.