[ofa-general] Re: QoS management in OpenSM - doc
Yevgeny Kliteynik
kliteyn at dev.mellanox.co.il
Sun Mar 2 01:39:15 PST 2008
Sasha Khapyorsky wrote:
> Hi Yevgeny,
>
> On 16:42 Wed 27 Feb , Yevgeny Kliteynik wrote:
>> The following doc describes QoS management in OpenSM.
>> This doc (named QoS_management_in_OpenSM.txt) has been added to
>> the OFED docs, along with the QoS_in_OFED.txt.
>>
>> I'd like to add this info to OpenSM man pages as well.
>
> Yes, I think that it could be useful to have it under opensm/doc too.
>
>> I'm including the text here as is, so it will be easier to follow
>> possible changes. When those will be done, I'll fix the format to
>> match the OpenSM man pages and post a patch.
>>
>> The only problem is that the whole OpenSM man has ~850 lines,
>> while this QoS management file has ~500 lines... :)
>
> I would suggest to have some basic part (50-100 lines) included in the
> man page and reference an entire document (under opensm/doc) for more
> details.
OK, I can prepare some kind of summary that would go into the man page.
However, this means that a user would no be able to define a QoS policy
just from reading an OpenSM man pages - he will HAVE to check the full
doc under opensm/doc.
>> Please review.
>
> Looks fine, few tiny nits are below.
>
> [snip...]
>
>> ==============================================================================
>> 4. Policy File Syntax Guidelines
>> ==============================================================================
>>
>> - Empty lines are ignored.
>
> It is mentioned on the next line too.
Right
>
>> - Leading and trailing blanks, as well as empty lines, are ignored, so
>> the indentation in the example is just for better readability.
>> - Comments are started with the pound sign (#) and terminated by EOL.
>> - Any keyword should be the first non-blank in the line, unless it's a
>> comment.
>> - Keywords that denote section/subsection start have matching closing
>> keywords.
>> - Having a QoS Level named "DEFAULT" is a must - it is applied to PR/MPR
>> requests that didn't match any of the matching rules.
>> - Any section/subsection of the policy file is optional.
>
> [snip...]
>
>> ==============================================================================
>> 6. Simplified QoS Policy - Details and Examples
>> ==============================================================================
>>
>> Simplified QoS policy match rules are tailored for matching ULPs (or some
>> application on top of a ULP) PR/MPR requests. This section has a list of
>> per-ULP (or per-application) match rules and the SL that should be enforced
>> on the matched PR/MPR query.
>>
>> Match rules include:
>> - Default match rule that is applied to PR/MPR query that didn't match any
>> of the other match rules
>> - SDP
>> - SDP application with a specific target TCP/IP port range
>> - SRP with a specific target IB port GUID
>> - RDS
>> - iSER
>> - iSER application with a specific target TCP/IP port range
>> - IPoIB with a default PKey
>> - IPoIB with a specific PKey
>> - any ULP/application with a specific Service ID in the PR/MPR query
>> - any ULP/application with a specific PKey in the PR/MPR query
>> - any ULP/application with a specific target IB port GUID in the PR/MPR
>> query
>>
>> Since any section of the policy file is optional, as long as basic rules of
>> the file are kept (such as no referring to nonexisting port group, having
>> default QoS Level, etc), the simplified policy section (qos-ulps) can serve
>> as a complete QoS policy file.
>> The shortest policy file in this case would be as follows:
>>
>> qos-ulps
>> default : 0 #default SL
>> end-qos-ulps
>>
>> It is equivalent to the previous example of the shortest policy file, and
>> it
>> is also equivalent to not having policy file at all.
>>
>> Below is an example of simplified QoS policy with all the possible
>> keywords:
>>
>> qos-ulps
>> default : 0 # default SL
>> sdp, port-num 30000 : 0 # SL for application running on
>> top
>> # of SDP when a destination
>> # TCP/IPport is 30000
>> sdp, port-num 10000-20000 : 0
>> sdp : 1 # default SL for any other
>> # application running on top of
>> SDP
>> rds : 2 # SL for RDS traffic
>> iser, port-num 900 : 0 # SL for iSER with a specific
>> target
>> # port
>> iser : 3 # default SL for iSER
>> ipoib, pkey 0x0001 : 0 # SL for IPoIB on partition with
>> # pkey 0x0001
>> ipoib : 4 # default IPoIB partition,
>> # pkey=0x7FFF
>> any, service-id 0x6234 : 6 # match any PR/MPR query with a
>> # specific Service ID
>> any, pkey 0x0ABC : 6 # match any PR/MPR query with a
>> # specific PKey
>> srp, target-port-guid 0x1234 : 5 # SRP when SRP Target is located
>> on
>> # a specified IB port GUID
>> any, target-port-guid 0x0ABC-0xFFFFF : 6 # match any PR/MPR query
>> with
>> # a specific target port GUID
>> end-qos-ulps
>
> Likely I missed this in implementation phase. But isn't it better to
> have ULPs to match QoS level rather than SL? Or probably both?
The goal of this part is to give up some flexibility in order to get maximum
simplicity. If an administrator wants to define a qos-level with more than
just SL (and I think that in most cases he wouldn't want to do that), there's
another way to refer to such qos-level - trough match rules.
>> Similar to the full policy definition, matching of PR/MPR queries is done
>> in
>> order of appearance in the QoS policy file such as the first match takes
>> precedence, except for the "default" rule, which is applied only if the
>> query
>> didn't match any other rule.
>>
>> All other sections of the QoS policy file take precedence over the qos-ulps
>> section. That is, if a policy file has both qos-match-rules and qos-ulps
>> sections, then any query is matched first against the rules in the
>> qos-match-rules section, and only if there was no match, the query is
>> matched
>> against the rules in qos-ulps section.
>>
>> Note that some of these match rules may overlap, so in order to use the
>> simplified QoS definition effectively, it is important to understand how
>> each
>> of the ULPs is matched:
>>
>> 6.1 IPoIB
>> IPoIB query is matched by PKey. Default PKey for IPoIB partition is 0x7fff,
>> so
>> the following three match rules are equivalent:
>>
>> ipoib : <SL>
>> ipoib, 0x7fff : <SL>
>> any, pkey 0x7fff : <SL>
>>
>> 6.2 SDP
>> SDP PR query is matched by Service ID. The Service-ID for SDP is
>> 0x000000000001PPPP, where PPPP are 4 hex digits holding the remote TCP/IP
>> Port
>> Number to connect to. The following two match rules are equivalent:
>>
>> sdp : <SL>
>> any, service-id 0x0000000000010000-0x000000000001ffff : <SL>
>>
>> 6.3 RDS
>> Similar to SDP, RDS PR query is matched by Service ID. The Service ID for
>> RDS
>> is 0x000000000106PPPP, where PPPP are 4 hex digits holding the remote
>> TCP/IP
>> Port Number to connect to. Default port number for RDS is 0x48CA, which
>> makes
>> a default Service-ID 0x00000000010648CA. The following two match rules are
>> equivalent:
>>
>> rds : <SL>
>> any, service-id 0x00000000010648CA : <SL>
>>
>> 6.4 iSER
>> Similar to RDS, iSER query is matched by Service ID, where the the Service
>> ID
>> is also 0x000000000106PPPP. Default port number for iSER is 0x035C, which
>> makes
>> a default Service-ID 0x000000000106035C. The following two match rules are
>> equivalent:
>>
>> iser : <SL>
>> any, service-id 0x000000000106035C : <SL>
>>
>> 6.5 SRP
>> Service ID for SRP varies from storage vendor to vendor, thus SRP query is
>> matched by the target IB port GUID. The following two match rules are
>> equivalent:
>>
>> srp, target-port-guid 0x1234 : <SL>
>> any, target-port-guid 0x1234 : <SL>
>>
>> Note that any of the above ULPs might contain target port GUID in the PR
>> query, so in order for these queries not to be recognized by the QoS
>> manager
>> as SRP, the SRP match rule (or any match rule that refers to the target
>> port
>> guid only) should be placed at the end of the qos-ulps match rules.
>>
>> 6.6 MPI
>> SL for MPI is manually configured by MPI admin. OpenSM is not forcing any
>> SL
>> on the MPI traffic, and that's why it is the only ULP that did not appear
>> in
>> the qos-ulps section.
>>
>>
>> ==============================================================================
>> 7. SL2VL Mapping and VL Arbitration
>> ==============================================================================
>
> I think here should be stated that both policies should be merged at
> some point.
Agree
-- Yevgeny
>
> [snip...]
>
> Sasha
>
More information about the general
mailing list