[ofa-general] IBV_WC_LOC_PROT_ERROR in receive

[Bharath Ramesh]

I checked for the following:
1) I havent deregistered the MR.
2) I am using a RC QP
3) The messages size are the same 40 bytes.
4) I only have one PD for the entire application, i.e both QP and MR
belong to the same PD
5) The vendor error that I get in the WC is error code 52.
6) I forgot to mention this in the earlier mail the snippet for my send
is as follows:

struct ibv_sge sge;
struct ibv_wc wc;
struct ibv_send_wr wr;
struct ibv_send_wr *wr_bad;

sge.addr = (uintptr_t) buf;
sge.length = size;
wr.wr_id = WR_ID;
wr.next = NULL;
wr.opcode = IBV_WR_SEND;
wr.send_flags = IBV_SEND_INLINE;
wr.num_sge = 1;
wr.sg_list = &sge;
if (ibv_post_send (ib_qp, &wr, &wr_bad) != 0) {
  printf ("ERROR: Unable to post send WR to queue.\n");
  return -1;
}

Thanks,

Bharath

* Dotan Barak (dotanb at dev.mellanox.co.il) wrote:
> Hi.
>
> Bharath Ramesh wrote:
>> I am getting this error when I am trying to do a bunch of send/receives.
>> I have registered the receive buffer. I printed the address of the
>> buffers and their respective lkeys, they all match but I am still
>> getting this error.
>>
>> The code snippet looks as follows:
>>
>> struct ibv_mr *mr;
>> struct ibv_sge sge;
>> struct ibv_recv_wr wr;
>> struct ibv_recv_wr *wr_bad;
>>
>> // registering buffers
>> mr = ibv_reg_mr (ib_pd, buf, size, IBV_ACCESS_LOCAL_WRITE |
>>                         IBV_ACCESS_REMOTE_READ |
>> 			IBV_ACCESS_REMOTE_WRITE);
>>
>>
>> //Post the receive buffer
>> sge.addr = (uintptr_t) buf;
>> sge.length = size;
>> sge.lkey = mr->lkey;
>> wr.wr_id = WR_ID;
>> wr.next = NULL;
>> wr.sg_list = &sge;
>> wr.num_sge = 1;
>> if (ibv_post_recv (ib_qp, &wr, &wr_bad) != 0) {
>>   printf ("ERROR: Unable to post receiver buffer.\n");
>>   return -1;
>> }
>>
>> When I poll for the completion event I get this error. Any help on this
>> is appreciated. I am not subscribed to this list, I would appreciate if
>> you please cc me on the reply.
>>   
>
>
> If the address that you given in the RR is valid (you didn't deregister 
> this MR):
> You should check the following things:
> * If this is a UD QP, maybe the extra 40 bytes (for the GRH) is missing in 
> the recv buffer.
> * Maybe the incoming message is larger than the receive buffer
> * maybe the PD of the QP and the MR are not the same
>
>
> If this didn't help you, the value of the vendor_err in the completion 
> structure may help me....
>
> Dotan
>

---
Bharath Ramesh       <bramesh at vt.edu>       http://people.cs.vt.edu/~bramesh