[ofa-general] ibv_post_send returns -1

[Bharath Ramesh]

Thanks Dotan, probably I didn't word my query correctly. It should have been
like is there a 1-1 mapping between the number of polls to send CQ and the
number of calls made to ibv_post_send. The only thing I think that can be
going wrong is that, I set the IBV_SEND_SIGNALED flag once in every 256
control messages. Which I assume implies I am getting a completion event
only for one of every 256 control messages. Hence the outstanding WR queue
is never cleared completely and I finally run out of WR that can be posted
to the SQ as large messages are also posted to the same SQ. I have 1024 WR
associated with the SQ. I poll the send CQ for every large message but this
is not true for every control message that I post to the SQ which might be
leading to a starvation of available WR's.

Regards,

Bharath


-----Original Message-----
From: Dotan Barak [mailto:dotanba at gmail.com] 
Sent: Monday, December 29, 2008 5:59 AM
To: Bharath Ramesh
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general] ibv_post_send returns -1

On Sat, Dec 27, 2008 at 10:17 PM, Bharath Ramesh <bramesh at vt.edu> wrote:
> Thanks for the reply Dotan, I am posting what I am doing currently,
probably
> I am missing something which you could point out to me.
>
> 1) I don't have completion notifications or completion channel associated
> with my send CQ.
> 2) I only have RC QPs.
> 3) Every few hundred messages (256 messages to be precise) for my small
> messages (64 bytes long) for which I use IBV_WR_SEND with IBV_SEND_INLINE
I
> set the IBV_SEND_SIGNALED flag. Once I set the flag I poll my send CQ for
> completion messages.
> 4) All my RDMA buffers for large messages are all registered, and I use
> IBV_SEND_SIGNALED flag for the WR and poll the send CQ for completion
> immediately after ibv_post_send.
>
> The only time when I don't poll my send CQ for completion for
ibv_post_send
> is for my small messages. I presume that polling the send CQ once every
256
> messages for these should be enough to free up the WR from the SQ for the
> previously posted WR. Am I wrong in assuming this, should each and every
> ibv_post_send should be associated with poll to the send CQ?
>
No, you are right. You can poll the CQ once in a while and not after
every post, but you have to make sure that the total number
of outstanding WR in the Send Queue is less that the value that the QP
was created with.

Dotan