[openib-general] QP sq->tail always = 0

[Fab Tillier]

> From: Jeff Carr [mailto:jcarr at linuxmachines.com]
> Sent: Thursday, May 26, 2005 3:56 PM
> 
> On 05/26/05 14:35, Sean Hefty wrote:
> > If the code is based on cmpost, this is likely the case.  Cmpost creates
> > the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED
flag.
> >
> > Since people are refering to this as a sample, I will look at changing
> > the code to avoid confusing people.
> 
> Another thing I noticed that seemed very strange to me, is that when I
> went back to test this problem with cmpost, the debugging output was
> different for the SQ and RQ.
> 
> cmpost.c has:
>          qp_attr->send_cq = node->cq;
>          qp_attr->recv_cq = node->cq;
>          node->qp = ib_create_qp(test.pd, qp_attr);
> 
> But, I would get:
> 
> ib_poll_cq == 1
> ib_mthca 0000:09:00.0: SQ 030406 (0 head, 0 tail, 5 max)
> ib_mthca 0000:09:00.0: RQ 030406 (5 head, 2 tail, 5 max
> 
> 			SQ->head != RQ->head
> 
> So this is kind of strange to me. I would think that the head & tail of
> the SQ and RQ should be the same. Maybe making the send and receive
> queue the same doesn't work correctly yet?

You're confusing QP depth and CQ depth.  The QP, being a pair, has both a
send and receive queue.  These have independent depths associated with them
- that is, the RQ and SQ can have different maximums.  Work requests are
posted independently to these two queues.  These queues are in effect sinks
- you feed requests into them, and once posted you don't get any feedback
from the QP related to their status.

The CQ is the source of completion information - this is where you find out
what completed.  You can have a multitude of RQs and SQs associated with a
single CQ.  When a completion happens on a queue pair, the completion
processing updates the indexes for the appropriate requests queue (SQ or RQ,
depending on the completion).  Because the RQ and SQ are independent, they
have independent head and tails.  The CQ is a single queue where all
completions get reported in FIFO order, with no segregation based on the
type of completion.  That is, the CQ doesn't have a SQ and RQ.

cmpost.c is just setting the same CQ for the SQ and RQ - this doesn't affect
theses queue's operation - it just sets where completions get reported.

I hope this makes sense.  I'll be glad to expand if I've confused you.

- Fab