[ofa-general] opensm routing

[Or Gerlitz]

Yevgeny Kliteynik wrote:
> Besides, if Lustre sends path queries to OpenSM (via RDMACM) when 
> opening their own QPs, then Lustre can use other SLs too. SM can even 
> assign different SLs to communication to Lustre metadata servers and 
> object storage servers if they are running on separate hosts, all 
> based on the path query.
Indeed, since Lustre uses the RDMA-CM which in turn does its path query 
in a <SGID, DGID, PKEY, SID, TOS> manner you can use either of the 
existing mechanisms in openSM to assign SL to this QP:

1. port group based (eg based on the SGID, DGID from the query)
2. pkey based (eg based on the PKEY)
3. SID based
4. TOS based
5. you name it...

~Everything happens under the cover of the rdma-cm, that is your app 
need to do nothing such that <SGID, DGID, PKEY, SID, TOS=0> would be 
provided by the rdma-cm in the path query, and the rdma-cm uses the SL 
from the returned path when your app calls rdma_create_qp. Only if you 
want non zero TOS to be in the query, rdma_set_service_type has to be 
called before rdma_resolve_route.

The approach is that the host provides the information, the IB sys admin 
design a QoS setting for the fabric and apply it through the mechanisms 
of the openSM, please see the cool presentation of Liran Liss from 
Sonoma @ 
http://www.openfabrics.org/archives/spring2008sonoma/Tuesday/qos_sonoma08_ofa_v1.ppt
and let us know if you have any further questions.

For MPIs which don't use path query the IB sys admin have instruct the 
users which SL to manually make the ranks of this job use, which is by 
far much less convenient.

Or.