[openib-general] QP sq->tail always = 0

[Jeff Carr]

On 05/26/05 14:35, Sean Hefty wrote:
> Roland Dreier wrote:
> 
>> Are you seeing any send completions on the CQ attached to the send
>> queue?  

No.

>> One possibility is that you're not generating any send
>> completions because you created your CQ with a sq_sig_type of
>> IB_SIGNAL_REQ_WR and then posting send requests without the
>> IB_SEND_SIGNALED flag.

Well, you seem to have hit the nail on the head.

>> (The IB_SIGNAL_REQ_WR flag means that not all send requests posted to
>> the send queue will generate a completion -- only those marked with
>> the IB_SEND_SIGNALED flag.  However, the driver can't free a send
>> request from the send queue until it knows it has completed, and the
>> only way for the driver to know that is to see a completion for the
>> given request or a later request.  Requests on a queue always complete
>> in order, so if a later request completes and generates a completion,
>> the driver can also free any earlier unsignaled requests)

Thanks very much for this description. It's certainly not trivial to 
understand such relationships. The IB spec 11.2.4.1 has:

· The Signaling Type must be specified for the Send Queue on
   this QP. The valid types are:

   ·   Non-selectable: All Work Requests submitted to the Send
         Queue always generate a completion entry.
   ·   Selectable: Consumer must specify on each Work
         Request submitted to the Send Queue whether to
         generate a completion entry for successful completions.

So, this sounds like the same thing and will read more from there. 
Thanks for sharing your understanding of IB.

> 
> If the code is based on cmpost, this is likely the case.  Cmpost creates 
> the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED flag.
> 
> Since people are refering to this as a sample, I will look at changing 
> the code to avoid confusing people.

Thanks; I'll try it out also and will let you know how it goes.

Jeff

BTW, IB_SIGNAL_REQ_WR is only referenced 3 times in the kernel without 
any comments. Boy it would be nice if some of the people that spent so 
much time on the IB arch spec spent some time submitting comments as 
patches to the Linux kernel sources. That would be a dream.

drivers/infiniband/ulp/sdp/sdp_conn.c:          init_attr->sq_sig_type 
   = IB_SIGNAL_REQ_WR;
drivers/infiniband/include/ib_verbs.h:  IB_SIGNAL_REQ_WR
drivers/infiniband/core/uverbs_cmd.c:   attr.sq_sig_type   = 
cmd.sq_sig_all ? IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR;