[openib-general] QP sq->tail always = 0

[Jeff Carr]

In my code (based on cmtest) I was using a single CQ for both send and 
recv. I broke them into seperate queues to try to figure out why I could 
only do N transfers for each CQ that I created.

I added some debugging output to mthca_cq.c to dump out the SQ and RQ 
for each QP.

When I run my test, the output for the server and client look similar.

For the server, it first runs: ib_post_recv() for N times (N=5 here)

ib_mthca 0000:09:00.0: SQ 050406 (1 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (2 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (3 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (4 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)

Then, the client sends N=5 responses via ib_post_recv() that the server 
sees via ib_poll_cq().

ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 1 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 2 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 3 tail, 5 max)
ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max)
ib_mthca 0000:09:00.0: RQ 050406 (5 head, 4 tail, 5 max)

ib_mthca 0000:09:00.0: RQ 050406 (5 head, 5 tail, 5 max
ib_mthca 0000:09:00.0: SQ 050406 full (5 head, 0 tail, 5 max, 0 nreq)

At this point, the RQ is empty: (head == tail)
And at this point the SQ is full: ((head - tail) == max)

Is this a correct understanding?

In both the server & client cases, the SQ always remains full. How are 
the connections supposed to be tagged as completed or pulled off of the 
queue?

Thanks for any help,
Jeff

PS: Some simple routines for debugging might be helpful for new users of 
infiniband.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mthca_qp.debug.diff
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050526/b5edcd8f/attachment.ksh>