[openib-general] openMPI for 2.6.17.10 kernel

[Jeff Squyres]

On Nov 30, 2006, at 1:30 PM, Steve Wise wrote:

>> Remember that OMPI currently does not use any CM for OF connections
>> at all.  So it's not like it's using the old CM that doesn't support
>> iWARP.  OMPI uses its own out-of-band mechanism, which, as I
>> understand it, should work with iWARP just as well as it works for  
>> IB.
>>
>> Am I incorrect in thinking that?  (I have no iWARP hardware to test
>> with)
>
> iWARP _requires_ the RDMA-CM for connection setup...

This is not what I understood from conversations in Tampa last week.

OMPI does not use CM at all today (RDMA CM or plain CM).  Instead,  
OMPI *only* uses its out-of-band connection to exchange information  
to then create queue pairs manually.  Leaving out all the details, it  
does something like this:

	// initiatior
	ibv_create_qp(...);
	ibv_modify_qp(...);// set pkey, port num, etc.
	send_via_oob_channel(lid, mtu, my_port_num, ...other info...);
	wait_for_oob_reply(...);
	ibv_modify_qp(...);//set mtu, lid, remote_port #, etc.

Are you telling me that the *ONLY* way to make QPs over iWARP  
connections is to use the RDMA CM, and it is not possible to work  
using a scheme like i listed above (passing relevant info via OOB  
channels)?  If so, I'm curious: what magic is the RDMA CM doing such  
that this is necessary?

-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems