[ofa-general] some comments/clarifications on ipoib/opensm re qos

[Yevgeny Kliteynik]

Hi Or,

Or Gerlitz wrote:
> Hi Yevgeny,
>
> I wanted to clarify to/with you some issues re QoS/IPoIB in openSM. 
> Looking in the documents provided by the ofed-docs package:
>
> from QoS_in_OFED.txt
>>  147 
>> ============================================================================== 
>>
>>  148 5. IPoIB
>>  149 
>> ============================================================================== 
>>
>>  151 IPoIB queries the SA for its broadcast group information.
>>  152 It provides the broadcast group SL, MTU, and RATE in every 
>> following
>>  153 PathRecord query performed when a new UDAV is needed by IPoIB.
> This is almost all wrong, sorry... the way it works in the Linux ipoib 
> driver, is the following:
>
> address vectors for unicast traffic, both datagram and connected 
> modes, are created through the result of an SA path query with this 
> comp mask <SGID, DGID, NUMB_PATH, TRAFFIC_CLASS, PKEY> where numb_path 
> is set to one, and the traffic class is the one returned by the SA to 
> the broadcast group, see below.
>
> address vectors for multicast senders only (clients) are created 
> through the result of an SA MC member query (join) with this comp mask 
> <MGID, SGID, PKEY, JOIN_STATE>
> address vectors for multicast receivers (that might send as well) are 
> created through the result of an SA MC member query (join) with this 
> comp mask <MGID, SGID, PKEY, JOIN_STATE>
>
> For all but the broadcast group, the following bits are also set in 
> the comp mask
> <QKEY, MTU_SELECTOR, MTU, TRAFFIC_CLASS, RATE_SELECTOR, RATE, SL, 
> FLOW_LABEL, HOP_LIMIT> where they are all derived from the broadcast 
> group, where in the broadcast case, the SA has assigned values for them.
>
> Bottom line, for path queries the --requested-- SL is not provided to 
> the SM and there's a difference between the info provided for sender 
> joins to server joins where only for the latter the driver asks for 
> specific <SL, RATE, MTU>.

OK. Can you rephrase what you said in two sentences? Something short
to replace the existing (and wrong, as you pointed out) three lines:

   151 IPoIB queries the SA for its broadcast group information.
   152 It provides the broadcast group SL, MTU, and RATE in every following
   153 PathRecord query performed when a new UDAV is needed by IPoIB.

>
> from QoS_management_in_opensm.txt
>>  353 6.1  IPoIB
>>  354 IPoIB query is matched by PKey. Default PKey for IPoIB partition 
>> is 0x7fff, so
>>  355 the following three match rules are equivalent:
>>  356
>>  357     ipoib            : <SL>
>>  358     ipoib, 0x7fff    : <SL>
>>  359     any, pkey 0x7fff : <SL>
> I see, so ipoib is just an acronym for a path query with the default 
> pkey.

"ipoib" w/o any pkey is an acronym for a path query with the default pkey,
but this rule also tells  OpenSM to  set the partition's SL to <SL>, and
to set the IPoIB mcast group's SL to <SL>.

Actually, line 358 is wrong (I'll update it). It should be as follows:
   357     ipoib              : <SL>
   358     ipoib, pkey 0x7fff : <SL>
   359     any, pkey 0x7fff   : <SL>

These three lines are equivalent.

>
> This creates confusion and I am not sure its well defined, at least in 
> my case when I put my hands on QoS testing, I couldn't guess that 
> "ipoib" == "pkey 0x7fff". For example, is it correct that "ipoib, pkey 
> 0x8001 : <SL>" is not valid rule?

This rule is also valid, providing that you actually have a partition
with pkey 0x8001 configured in the partition cfg file.

-- Yevgeny

>
> Or.
>
>
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>