[ofa-general] some comments/clarifications on ipoib/opensm re qos

[Or Gerlitz]

Hi Yevgeny,

I wanted to clarify to/with you some issues re QoS/IPoIB in openSM. 
Looking in the documents provided by the ofed-docs package:

from QoS_in_OFED.txt
>  147 
> ==============================================================================
>  148 5. IPoIB
>  149 
> ==============================================================================
>  151 IPoIB queries the SA for its broadcast group information.
>  152 It provides the broadcast group SL, MTU, and RATE in every following
>  153 PathRecord query performed when a new UDAV is needed by IPoIB.
This is almost all wrong, sorry... the way it works in the Linux ipoib 
driver, is the following:

address vectors for unicast traffic, both datagram and connected modes, 
are created through the result of an SA path query with this comp mask 
<SGID, DGID, NUMB_PATH, TRAFFIC_CLASS, PKEY> where numb_path is set to 
one, and the traffic class is the one returned by the SA to the 
broadcast group, see below.

address vectors for multicast senders only (clients) are created through 
the result of an SA MC member query (join) with this comp mask <MGID, 
SGID, PKEY, JOIN_STATE> 

address vectors for multicast receivers (that might send as well) are created through the result of an SA MC member query (join) with this comp mask <MGID, SGID, PKEY, JOIN_STATE>

For all but the broadcast group, the following bits are also set in the comp mask
<QKEY, MTU_SELECTOR, MTU, TRAFFIC_CLASS, RATE_SELECTOR, RATE, SL, FLOW_LABEL, HOP_LIMIT> where they are all derived from the broadcast group, where in the broadcast case, the SA has assigned values for them.

Bottom line, for path queries the --requested-- SL is not provided to the SM and there's a difference between the info provided for sender joins to server joins where only for the latter the driver asks for specific <SL, RATE, MTU>.

from QoS_management_in_opensm.txt 

>  353 6.1  IPoIB
>  354 IPoIB query is matched by PKey. Default PKey for IPoIB partition 
> is 0x7fff, so
>  355 the following three match rules are equivalent:
>  356
>  357     ipoib            : <SL>
>  358     ipoib, 0x7fff    : <SL>
>  359     any, pkey 0x7fff : <SL>
I see, so ipoib is just an acronym for a path query with the default pkey.

This creates confusion and I am not sure its well defined, at least in 
my case when I put my hands on QoS testing, I couldn't guess that 
"ipoib" == "pkey 0x7fff". For example, is it correct that "ipoib, pkey 
0x8001 : <SL>" is not valid rule?

Or.