[ofa-general] ibv_post_send returns -1

[Bharath Ramesh]

Thanks for the reply Dotan, I am posting what I am doing currently, probably
I am missing something which you could point out to me.

1) I don't have completion notifications or completion channel associated
with my send CQ.
2) I only have RC QPs.
3) Every few hundred messages (256 messages to be precise) for my small
messages (64 bytes long) for which I use IBV_WR_SEND with IBV_SEND_INLINE I
set the IBV_SEND_SIGNALED flag. Once I set the flag I poll my send CQ for
completion messages.
4) All my RDMA buffers for large messages are all registered, and I use
IBV_SEND_SIGNALED flag for the WR and poll the send CQ for completion
immediately after ibv_post_send.

The only time when I don't poll my send CQ for completion for ibv_post_send
is for my small messages. I presume that polling the send CQ once every 256
messages for these should be enough to free up the WR from the SQ for the
previously posted WR. Am I wrong in assuming this, should each and every
ibv_post_send should be associated with poll to the send CQ?

Regards,

Bharath


-----Original Message-----
From: Dotan Barak [mailto:dotanba at gmail.com] 
Sent: Saturday, December 27, 2008 1:50 AM
To: Bharath Ramesh
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general] ibv_post_send returns -1

Bharath Ramesh wrote:
> I using OFED-1.2 for my research project. I am facing an issue where 
> when I try to post IBV_WR_RDMA_WRITE work request to the QP 
> ibv_post_send returns with -1. This I presume means lack of resources 
> to complete the operation. To give a brief summary of my communication 
> mechanism, I have two kind of messages control messages and large 
> messages. I post control messages with the normal IBV_WR_SEND with 
> send flags set to IBV_SEND_INLINE. Every few hundred such control 
> messages I set the send flag to IBV_SEND_INLINE | IBV_SEND_SIGNALED 
> and poll my send CQ. For large messages I use IBV_WR_RDMA_WRITE with 
> send flags set to IBV_SEND_SIGNALED. After posting any large message I 
> poll the send CQ for completion. I have sufficient buffers posted on 
> the receiver.
>
> The first thing I did when I encountered this issue is to write a 
> small test which just does only IBV_WR_RDMA_WRITE similar to my 
> application. This test works fine without having any issues. I am 
> wondering what could be potential issues that might be causing such a 
> scenario of unavailable resources. Any help in debugging this 
> situation would be appreciated. I am not subscribed to the list would 
> appreciate if I am copied on the replies.
Hi.


ibv_post_send may fail if:
1) too many WR are outstanding on the SQ (maybe you posted too many WR 
without polling for there completion)
2) bad WR was posted (for example: RDMA Write for UD QP).
3) maybe the size of the WR that you are trying to post as inline is too 
big (upon QP creation you can see this size in the sq_send_inline)


Dotan