[ofa-general] Re: QoS management in OpenSM - doc
Sasha Khapyorsky
sashak at voltaire.com
Sat Mar 1 13:01:32 PST 2008
Hi Yevgeny,
On 16:42 Wed 27 Feb , Yevgeny Kliteynik wrote:
>
> The following doc describes QoS management in OpenSM.
> This doc (named QoS_management_in_OpenSM.txt) has been added to
> the OFED docs, along with the QoS_in_OFED.txt.
>
> I'd like to add this info to OpenSM man pages as well.
Yes, I think that it could be useful to have it under opensm/doc too.
> I'm including the text here as is, so it will be easier to follow
> possible changes. When those will be done, I'll fix the format to
> match the OpenSM man pages and post a patch.
>
> The only problem is that the whole OpenSM man has ~850 lines,
> while this QoS management file has ~500 lines... :)
I would suggest to have some basic part (50-100 lines) included in the
man page and reference an entire document (under opensm/doc) for more
details.
> Please review.
Looks fine, few tiny nits are below.
[snip...]
> ==============================================================================
> 4. Policy File Syntax Guidelines
> ==============================================================================
>
> - Empty lines are ignored.
It is mentioned on the next line too.
> - Leading and trailing blanks, as well as empty lines, are ignored, so
> the indentation in the example is just for better readability.
> - Comments are started with the pound sign (#) and terminated by EOL.
> - Any keyword should be the first non-blank in the line, unless it's a
> comment.
> - Keywords that denote section/subsection start have matching closing
> keywords.
> - Having a QoS Level named "DEFAULT" is a must - it is applied to PR/MPR
> requests that didn't match any of the matching rules.
> - Any section/subsection of the policy file is optional.
[snip...]
> ==============================================================================
> 6. Simplified QoS Policy - Details and Examples
> ==============================================================================
>
> Simplified QoS policy match rules are tailored for matching ULPs (or some
> application on top of a ULP) PR/MPR requests. This section has a list of
> per-ULP (or per-application) match rules and the SL that should be enforced
> on the matched PR/MPR query.
>
> Match rules include:
> - Default match rule that is applied to PR/MPR query that didn't match any
> of the other match rules
> - SDP
> - SDP application with a specific target TCP/IP port range
> - SRP with a specific target IB port GUID
> - RDS
> - iSER
> - iSER application with a specific target TCP/IP port range
> - IPoIB with a default PKey
> - IPoIB with a specific PKey
> - any ULP/application with a specific Service ID in the PR/MPR query
> - any ULP/application with a specific PKey in the PR/MPR query
> - any ULP/application with a specific target IB port GUID in the PR/MPR
> query
>
> Since any section of the policy file is optional, as long as basic rules of
> the file are kept (such as no referring to nonexisting port group, having
> default QoS Level, etc), the simplified policy section (qos-ulps) can serve
> as a complete QoS policy file.
> The shortest policy file in this case would be as follows:
>
> qos-ulps
> default : 0 #default SL
> end-qos-ulps
>
> It is equivalent to the previous example of the shortest policy file, and
> it
> is also equivalent to not having policy file at all.
>
> Below is an example of simplified QoS policy with all the possible
> keywords:
>
> qos-ulps
> default : 0 # default SL
> sdp, port-num 30000 : 0 # SL for application running on
> top
> # of SDP when a destination
> # TCP/IPport is 30000
> sdp, port-num 10000-20000 : 0
> sdp : 1 # default SL for any other
> # application running on top of
> SDP
> rds : 2 # SL for RDS traffic
> iser, port-num 900 : 0 # SL for iSER with a specific
> target
> # port
> iser : 3 # default SL for iSER
> ipoib, pkey 0x0001 : 0 # SL for IPoIB on partition with
> # pkey 0x0001
> ipoib : 4 # default IPoIB partition,
> # pkey=0x7FFF
> any, service-id 0x6234 : 6 # match any PR/MPR query with a
> # specific Service ID
> any, pkey 0x0ABC : 6 # match any PR/MPR query with a
> # specific PKey
> srp, target-port-guid 0x1234 : 5 # SRP when SRP Target is located
> on
> # a specified IB port GUID
> any, target-port-guid 0x0ABC-0xFFFFF : 6 # match any PR/MPR query
> with
> # a specific target port GUID
> end-qos-ulps
Likely I missed this in implementation phase. But isn't it better to
have ULPs to match QoS level rather than SL? Or probably both?
> Similar to the full policy definition, matching of PR/MPR queries is done
> in
> order of appearance in the QoS policy file such as the first match takes
> precedence, except for the "default" rule, which is applied only if the
> query
> didn't match any other rule.
>
> All other sections of the QoS policy file take precedence over the qos-ulps
> section. That is, if a policy file has both qos-match-rules and qos-ulps
> sections, then any query is matched first against the rules in the
> qos-match-rules section, and only if there was no match, the query is
> matched
> against the rules in qos-ulps section.
>
> Note that some of these match rules may overlap, so in order to use the
> simplified QoS definition effectively, it is important to understand how
> each
> of the ULPs is matched:
>
> 6.1 IPoIB
> IPoIB query is matched by PKey. Default PKey for IPoIB partition is 0x7fff,
> so
> the following three match rules are equivalent:
>
> ipoib : <SL>
> ipoib, 0x7fff : <SL>
> any, pkey 0x7fff : <SL>
>
> 6.2 SDP
> SDP PR query is matched by Service ID. The Service-ID for SDP is
> 0x000000000001PPPP, where PPPP are 4 hex digits holding the remote TCP/IP
> Port
> Number to connect to. The following two match rules are equivalent:
>
> sdp : <SL>
> any, service-id 0x0000000000010000-0x000000000001ffff : <SL>
>
> 6.3 RDS
> Similar to SDP, RDS PR query is matched by Service ID. The Service ID for
> RDS
> is 0x000000000106PPPP, where PPPP are 4 hex digits holding the remote
> TCP/IP
> Port Number to connect to. Default port number for RDS is 0x48CA, which
> makes
> a default Service-ID 0x00000000010648CA. The following two match rules are
> equivalent:
>
> rds : <SL>
> any, service-id 0x00000000010648CA : <SL>
>
> 6.4 iSER
> Similar to RDS, iSER query is matched by Service ID, where the the Service
> ID
> is also 0x000000000106PPPP. Default port number for iSER is 0x035C, which
> makes
> a default Service-ID 0x000000000106035C. The following two match rules are
> equivalent:
>
> iser : <SL>
> any, service-id 0x000000000106035C : <SL>
>
> 6.5 SRP
> Service ID for SRP varies from storage vendor to vendor, thus SRP query is
> matched by the target IB port GUID. The following two match rules are
> equivalent:
>
> srp, target-port-guid 0x1234 : <SL>
> any, target-port-guid 0x1234 : <SL>
>
> Note that any of the above ULPs might contain target port GUID in the PR
> query, so in order for these queries not to be recognized by the QoS
> manager
> as SRP, the SRP match rule (or any match rule that refers to the target
> port
> guid only) should be placed at the end of the qos-ulps match rules.
>
> 6.6 MPI
> SL for MPI is manually configured by MPI admin. OpenSM is not forcing any
> SL
> on the MPI traffic, and that's why it is the only ULP that did not appear
> in
> the qos-ulps section.
>
>
> ==============================================================================
> 7. SL2VL Mapping and VL Arbitration
> ==============================================================================
I think here should be stated that both policies should be merged at
some point.
[snip...]
Sasha
More information about the general
mailing list