[ofa-general] What does IBV_WC_REM_OP_ERR after a verb send indicate?

[Nitin Mehrotra]

Folks,

I am getting this error on a verb send operation and I can't figure out 
what could be the cause; I searched for all instances of this error in 
the IB code and while I found 4, none was illuminating.

As a background, we are developing an IB application that uses RDMA for 
connection set up and the verbs interface for data transfer. We have 
tested the two ends as user space applications and they work - they can 
connect and exchange data. We are now converting the server end into a 
kernel module and this error is being encountered on the client when it 
posts a send to the RDMA connected QP. I have verified that the 
connection is setup and that recv WR with buffers are posted on the QP. 
Could it be a protection domain problem? Because we have multiple 
clients that connect to the one server we create the PD on the rdma_id 
that is used to connect to the server not the one that connection event 
gives us. Could that be the problem? We assume that the PD is tied to 
the IB device and there is only physical IB port in the system. If that 
is the problem, then why does this work in the userspace version and 
fail in the module version.

Appreciate any pointers on this.

Thanks,

Nitin