[ofa-general] Question about RDMA CM

[Jeff Squyres]

On Sep 17, 2008, at 1:25 PM, Jeff Squyres wrote:

>> We do not use rdma_get_devices() because our wireup scheme is both  
>> modular (it may use RDMA CM or it may use something else) and  
>> separate from the RDMA device discovery process.  Specifically, the  
>> PD that we have allocated is from the context we got back from  
>> ibv_open_device(), not rdma_get_devices().


To followup for the list -- preliminary testing seems to indicate that  
this was exactly the problem.  To create QPs with RDMA CM, one must  
use rdma_create_qp(), not ibv_create_qp(). Once I started doing this,  
my problems seem to have disappeared.

Jon M. quickly glanced at the RDMA CM kernel code and didn't see  
anything that jumped out at him that would matter on the iWARP side  
(i.e., using rdma_create_qp() vs. ibv_create_qp()).  But I assume that  
it *does* matter somehow for IB -- this would explain the behavior we  
saw (worked on iWARP; failed on IB).  Regardless, rdma_create_qp() in  
the RDMA CM interface, and therefore we should (and will) use it when  
using RDMA CM.

Thanks for the insight; it's exactly what I was looking for.

-- 
Jeff Squyres
Cisco Systems