[openib-general] APM support in openib stack

[Or Gerlitz]

Tang, Changqing wrote:
> Sorry, I find the function  'ib_sa_path_rec_get()'  in kernel code. Then
> here is my question:
> 
> Is there any way (instruction) to fill in struct 'ib_sa_path_rec' inside
> struct 'ib_cm_req_param' without using librdmacm.so ?

Hi CQ,

I understand that you considering to go on an approach which does not 
involve librdmacm, so you would probably like to

	+ use IB PORT GIDs at your initial mpi init exchange
	+ issue IB SA Path query via libibsa (which does not exist)
	+ establish IB RC connection (listen/connect/accept) via libibcm

Please note that such an approach is possible even with the non 
existence of libibsa (similarly to what IB MPIs do today use IB PORT 
LIDs [note you would need both GIDs & LIDs to have the CM working fine] 
at your ranks pre exchange and hardcode the other IB PATH params such as 
MTU, PKEY and SL you set later into the IB RC QP.

However, a no rdma cm approach means you need to apply hacks to "guess" 
the correct pkey, mtu and sl and some more limitations that eventually 
you would face when coming to advanced IB deployment environments.

All you have to do to use the rdma cm to require a functional IPoIB NIC 
on each of the active IB PORTs, which is a trivial requirement from the 
users in the Ethernet world, so why not apply it here as well ???

On the other hand moving to use the IB CM instead of emulating it via 
TCP is some progress...

Or.