<html>
<body>
<font size=3><br>
High-level feedback:<br><br>
- An IB fabric could be used for a single ULP and still require
QoS. The issue is how to differentiate flows on a given shared
element within the fabric.<br><br>
- QoS controls must be dynamic. The document references initialization as
the time when decisions are made but obviously that is just a first pass
on use of the fabric and not what it will become in potentially a short
period of time.<br><br>
- QoS also involves multi-path support (not really touched upon in terms
of specifics in this document). Distributing or segregating work
even if for the same ULP should be done across multiple or distinct
paths. In one sense this may complicate the work but in another it
is simpler in that arbitration controls for shared links become easier to
manage if the number of flows is reduced. <br><br>
- IP over IB defines a multicast group which is ultimately a spanning
tree. That should not constrain what paths are used to communicate
between endnode pairs. That only defines the multicast paths which
are not strongly ordered relative to the unicast traffic. Further
IP over IB may operate using the RC mode between endnodes. It is
very simple to replicate RC and then segregate these into QoS domains
(one could just align priority with the 802.1p for simplicity and
practical execution) which can in turn flow over shared or distinct
paths.<br><br>
- IB is a centrally managed fabric. Adding in SID into records and
such really isn't going to help solve the problem unless there is also a
centralized management entity well above IB that can prioritize
communication service rates for different ULP and endnode pairs.
Given most of these centralized management entities are rather ignorant
of IB at the moment, this presents a chicken-egg dilemma which is further
complicated by developing SOA technology. It might be more valuable
in one sense to examine SOA technology and how it is translating itself
to say Ethernet and then see how this can be leveraged to IB.<br><br>
- QoS needs to examine the sums of the consumers of a given path and
their service rate requirements. It isn't just about setting a
priority level but also about the packet injection rate to the fabric on
that priority. This needs to be taken into account as
well.<br><br>
Overall, it is not clear to me what the end value of this document.
The challenge for any network admin is to translate SOA driven
requirements into fabric control knob setting. Without such
translation algorithms / understanding, it is not clear that there is
anything truly missing in the IBTA spec suite or that this RFC will
really advance the integration of IB into the data center in a truly
meaningful manner.<br><br>
Mike<br><br>
<br>
At 07:53 AM 5/30/2006, Eitan Zahavi wrote:<br>
<blockquote type=cite class=cite cite="">To: OPENIB
<openib-general@openib.org><br>
Subject: QoS RFC - Resend using a friendly mailer<br>
--text follows this line--<br>
Hi All <br><br>
Please find the attached RFC describing how QoS policy support could be
implemented in the OpenFabrics stack.<br>
Your comments are welcome.<br><br>
Eitan<br><br>
RFC: OpenFabrics Enhancements for QoS Support<br>
===============================================<br><br>
Authors: . Eitan Zahavi <eitan@mellanox.co.il><br>
Date: .... May 2006.<br>
Revision: 0.1<br><br>
Table of contents:<br>
1. Overview<br>
2. Architecture<br>
3. Supported Policy<br>
4. CMA functionality<br>
5. IPoIB functionality<br>
6. SDP functionality<br>
7. SRP functionality<br>
8. iSER functionality<br>
9. OpenSM functionality<br><br>
1. Overview<br>
------------<br>
Quality of Service requirements stem from the realization of I/O
consolidation <br>
over IB network: As multiple applications and ULPs share the same fabric,
means <br>
to control their use of the network resources are becoming a must. The
basic <br>
need is to differentiate the service levels provided to different traffic
flows. <br>
Such that a policy could be enforced and control each flow utilization of
the <br>
fabric resources.<br><br>
IBTA specification defined several hardware features and management
interfaces <br>
to support QoS:<br>
* Up to 15 Virtual Lanes (VL) could carry traffic in a non-blocking
manner<br>
* Arbitration between traffic of different VL is performed by a 2
priority <br>
levels weighted round robin arbiter. The arbiter is programmable
with <br>
a sequence of (VL, weight) pairs and maximal number of high
priority credits <br>
to be processed before low priority is served<br>
* Packets carry class of service marking in the range 0 to 15 in
their<br>
header SL field<br>
* Each switch can map the incoming packet by its SL to a particular
output<br>
VL based on programmable table VL=SL-to-VL-MAP(in-port, out-port,
SL)<br>
* The Subnet Administrator controls each communication flow
parameters<br>
by providing them as a response to Path Record query<br><br>
The IB QoS features provide the means to implement a DiffServ like
architecture. <br>
DiffServ architecture (IETF RFC2474 2475) is widely used today in highly
dynamic <br>
fabrics. <br><br>
This proposal provides the detailed functional definition for the various
<br>
software elements that are required to enable a DiffServ like
architecture over <br>
the OpenFabrics software stack.<br><br>
<br><br>
<br><br>
2. Architecture<br>
----------------<br>
This proposal split the QoS functionality between the SM/SA, CMA and the
various <br>
ULPS. We take the "chronology approach" to describe how the
overall system <br>
works:<br><br>
2.1. The network manager (human) provides a set of rules (policy) that
defines <br>
how the network is being configured and how its resources are split to
different <br>
QoS-Levels. The policy also define how to decide which QoS-Level each
<br>
application or ULP or service use.<br><br>
2.2. The SM analyzes the provided policy to see if it is realizable and
performs <br>
the necessary fabric setup. The SM may continuously monitor the policy
and adapt <br>
to changes in it. Part of this policy defines the default QoS-Level of
each <br>
partition. The SA is being enhanced to match the requested Source,
Destination, <br>
TClass, Service-ID (and optionally SL and priority) against the policy.
So <br>
clients (ULPs, programs) can obtain a policy enforced QoS. The SM is also
<br>
enhanced to support setting up partitions with appropriate IPoIB
broadcast <br>
group. This broadcast group carries its QoS attributes: TClass, SL, MTU
and <br>
RATE.<br><br>
2.3. IPoIB is being setup. IPoIB uses the SL, MTU and RATE available on
the <br>
multicast group which forms the broadcast group of this
partition.<br><br>
2.4. MPI which provides non IB based connection management should be
configured <br>
to run using hard coded SLs. It uses these SLs in every QP being
opened.<br><br>
2.5. ULPs that use CM interface (like SRP) should have their own
pre-assigned <br>
Service-ID and use it while obtaining PathRecord for establishing their
<br>
connections. The SA receiving the PathRecord should match it against the
policy <br>
and return the appropriate PathRecord including SL, MTU, RATE and TClass.
<br><br>
2.6. ULPs and programs using CMA to establish RC connection should
provide the <br>
CMA the target IP and Service-ID. Some of the ULPs might also provide
TClass <br>
(E.g. for SDP sockets that are provided the TOS socket option). The CMA
should <br>
then use the provided Service-ID and optional TClass and pass them in the
<br>
PathRecord request. The resulting PathRecord should be used for
configuring the <br>
connection QP.<br><br>
PathRecord and MultiPathRecord enhancement for QoS: <br>
As mentioned above the PathRecord and MultiPathRecord attributes should
be <br>
enhanced to carry the Service-ID which is a 64bit value. Given the
existing <br>
definition for these attributes we propose to use the following fields
for <br>
Service-ID:<br>
* For PathRecord: use the first 2 reserved fields whicg are 32bits each
<br>
(component masks 0x1 and 0x2). Component mask 1 should be used to
refer to the <br>
merged Service-ID field<br>
* For MultiPathRecord: use 2 reserved fields: <br>
1. after the packet life (8 bits) which is component mask bit
0x10000 (17)<br>
2. the field before SDGID1 (56 bits) which is component mask bit
0x200000 (22)<br>
Once merged they should be selected using component mask bit
0x10000 (17)<br>
A new capability bit should describe the SM QoS support in the SA class
port <br>
info. This approach provides an easy migration path for existing access
layer <br>
and ULPs by not introducing a new attribute.<br><br>
<br>
3. Supported Policy<br>
-------------------- <br><br>
The QoS policy supported by this proposal is divided into 4 sub
sections:<br><br>
* Node Group: a set of HCAs, Routers or Switches that share the same
settings. <br>
A node groups might be a partition defined by the partition manager
policy in <br>
terms of GUIDs. Future implementations might provide support for
NodeDescription <br>
based definition of node groups.<br><br>
* Fabric Setup: <br>
Defines how the SL2VL and VLArb tables should be setup. This policy
definition <br>
assumes the computation of target behavior should be performed outside of
<br>
OpenSM.<br><br>
* QoS-Levels Definition:<br>
This section defines the possible sets of parameters for QoS that a
client might <br>
be mapped to. Each set holds: SL and optionally: Max MTU, Max Rate, Path
Bits <br>
(in case LMC > 0 is used for QoS) and TClass.<br><br>
* Matching Rules:<br>
A list of rules that match an incoming PathRecord request to a QoS-Level.
The <br>
rules are processed in order such as the first match is applied. Each
rule is <br>
built out of set of match expressions which should all match for the rule
to <br>
apply. The matching expressions are defined for the following fields<br>
** SRC and DST to lists of node groups<br>
** Service-ID to a list of Service-ID or Service-ID ranges<br>
** TClass to a list of TClass values or ranges<br><br>
XML style syntax is provided for the policy file. However, a strict BNF
format <br>
(provided in section 8) should be used for parsing it.<br><br>
<?xml version="1.0"
encoding="ISO-8859-1"?><br>
<qos-policy><br>
<!-- Port Groups define sets of ports to be used later in the
settings --><br>
<port-groups><br>
<!-- using port GUIDs --><br>
<port-group> <name>Storage</name> <use>our
SRP storage targets</use><br>
<port-guid>0x1000000000000001</port-guid><br>
<port-guid>0x1000000000000002</port-guid><br>
</port-group><br>
<!-- using names obtained by concatenation of first 2 words of
NodeDescription<br>
and port number --><br>
<port-group> <name>Virtual Servers</name>
<use>node desc and IB port #</use><br>
<port-name>vs1/HCA-1/P1</port-name><br>
<port-name>vs3/HCA-1/P1</port-name><br>
<port-name>vs3/HCA-2/P1</port-name><br>
</port-group><br>
<!-- using partitions defined in the partition policy
--><br>
<port-group> <name>Partition 1</name>
<use>default settings</use><br>
<partition>Part1</partition> <br>
</port-group><br>
<!-- using node types HCA|ROUTER|SWITCH --><br>
<port-group> <name>Routers</name> <use>all
routers</use><br>
<node-type>ROUTER</node-type> <br>
</port-group> <br>
</port-groups><br>
<qos-setup><br>
<!-- define all types of SL2VL tables always have 16 VL entries
--><br>
<sl2vl-tables><br>
<!-- scope defines the exact devices and in/out ports the
tables apply to<br>
if the same port is matching several rules the last
one applies --><br>
<sl2vl-scope> <group>Part1</group>
<from>*</from> <to>*</to> <br>
<sl2vl-table>0,1,2,3,4,5,6,7,8,9,10,11,12,13,14</sl2vl-table><br>
</sl2vl-scope><br>
<!-- "across" means the port just connected to
the given group, <br>
also the link across port 1 is probably
supporting only 2 VLs --><br>
<sl2vl-scope> <across>Storage</across>
<from>*</from> <to>1</to><br>
<sl2vl-table>0,1,1,1,1,1,1,1,1,1,1,1,1,1,1</sl2vl-table><br>
</sl2vl-scope><br>
<sl2vl-tables><br><br>
<!-- define all types of VLArb tables. The length of the tables
should <br>
match the physically supported tables by their target ports
--><br>
<vlarb-tables><br>
<!-- scope defines the exact ports the VLArb tables apply
to --><br>
<vlarb-scope> <group>Storage</group>
<to>*</to><br>
<!-- VLArb table holds VL and weight pairs
--><br>
<vlarb-high>0:255,1:127,2:63,3:31,4:15,5:7,6:3,7:1</vlarb-high><br>
<vlarb-low>8:255,9:127,10:63,11:31,12:15,13:7,14:3</vlarb-low><br>
<vl-high-limit>10</vl-high-limit><br>
</vlarb-scope><br>
</vlarb-tables><br>
</qos-setup><br><br>
<qos-levels><br>
<!-- the first one is just setting SL --><br>
<qos-level> <sn>1</sn> <use>for the lowest
priority comm</use><br>
<sl>16</sl><br>
</qos-level><br>
<!-- the second sets SL and TClass --><br>
<qos-level> <sn>2</sn> <use>low latency
best bandwidth</use><br>
<sl>0</sl>
<tclass>7</tclass><br>
</qos-level><br>
<!-- the whole set: SL, TClass, MTU-Limit, Rate-Limit,
Path-Bits --><br>
<qos-level> <sn>3</sn> <use>just an
example</use><br>
<sl>0</sl> <tclass>32</tclass>
<mtu_limit>1</mtl_limit> <br>
<rate_limit>1</rate_limit><br>
</qos-level><br>
</qos-levels><br><br>
<qos_match_rules><br>
<!-- matching by single criteria: tclass (list of values and
ranges) --><br>
<qos_match_rule> <sn>1</sn> <use>low
latency by tclass 7-9 or 11></use><br>
<tclass>7-9,11</tclass>
<match-level>1</match-level><br>
</qos_match_rule><br>
<!-- show matching by destination group AND service-ids
--><br>
<qos_match_rule> <sn>2</sn> <use>Storage
targets connection></use><br>
<destination>Storage</destination>
<service>22,4719</service><br>
<match-level>3</match-level><br>
</qos_match_rule><br>
</qos_match_rules><br><br>
</qos-policy><br><br>
<br>
4. IPoIB<br>
---------<br><br>
IPoIB already query the SA for its broadcast group information. The
additional <br>
functionality required is for IPoIB to provide the broadcast group SL,
MTU, RATE <br>
and TClass in every following PathRecord query performed when a new UDAV
is <br>
needed by IPoIB. <br>
We could assign a special Service-ID for IPoIB use but since all
communication <br>
on the same IPoIB interface shares the same QoS-Level without the ability
to <br>
differentiate it by target service we can ignore it for
simplicity.<br><br>
5. CMA features<br>
----------------<br><br>
The CMA interface supports Service-ID through the notion of port space as
a <br>
prefixes to the port_num which is part of the sockaddr provided to <br>
rdma_resolve_add(). What is missing is the explicit request for a TClass
that <br>
should allow the ULP (like SDP) to propagate a specific request for a
class of <br>
service. A mechanism for providing the TClass is available in the IPv6
address, <br>
so we could use that address field. Another option is to implement a
special <br>
connection options API for CMA.<br><br>
Missing functionality by CMA is the usage of the provided TClass and
Service-ID <br>
in the sent PathRecord. When a response is obtained it is an existing
<br>
requirement for the CMA to use the PathRecord from the response in
setting up <br>
the QP address vector.<br><br>
<br>
6. SDP<br>
-------<br><br>
SDP uses CMA for building its connections. <br>
The Service-ID for SDP is 0x000000000001PPPP, where PPPP are 4 hex digits
<br>
holding the remote TCP/IP Port Number to connect to.<br>
SDP might be provided with SO_PRIORITY socket option. In that case the
value <br>
provided should be sent to the CMA as the TClass option of that
connection. <br><br>
7. SRP<br>
-------<br><br>
Current SRP implementation uses its own CM callbacks (not CMA). So SRP
should <br>
fill in the Service-ID in the PathRecord by itself and use that
information in <br>
setting up the QP. The T10 SRP standard defines the SRP Service-ID to be
defined <br>
by the SRP target I/O Controller (but they should also comply with IBTA
Service-<br>
ID rules). Anyway, the Service-ID is reported by the I/O Controller in
the <br>
ServiceEntries DMA attribute and should be used in the PathRecord if the
SA <br>
reports its ability to handle QoS PathRecords.<br><br>
8. iSER<br>
--------<br>
iSER uses CMA and thus should be very close to SDP. The Service-ID for
iSER <br>
should be TBD.<br><br>
<br>
9. OpenSM features<br>
-------------------<br>
The QoS related functionality to be provided by OpenSM can be split into
two <br>
main parts:<br><br>
3.1. Fabric Setup<br>
During fabric initialization the SM should parse the policy and apply its
<br>
settings to the discovered fabric elements. The following actions should
be <br>
performed:<br>
* Parsing of policy<br>
* Node Group identification. Warning should be provided for each node not
<br>
specified but found.<br>
* SL2VL settings validation should be checked:<br>
+ A warning will be provided if there are no matching targets for
the SL2VL <br>
setting statement. <br>
+ An error message will be printed to the log file if an invalid
setting is <br>
found. A setting is invalid if it refers to:<br>
- Non existing port numbers of the target devices<br>
- Unsupported VLs for the target device. In the later
case the map to non<br>
existing VLs should be replaced to VL15
i.e. packets will be dropped.<br>
* SL2VL setting is to be performed<br>
* VL Arbitration table settings should be validated according to the
following <br>
rules:<br>
+ A warning will be provided if there are no matching targets for
the setting <br>
statement<br>
+ An error will be provided if the port number exceeds the target
ports<br>
+ An error will be generated if the table length exceeds device
capabilities<br>
+ An warning will be generated if the table quote a VL that is not
supported <br>
by the target device<br>
* VL Arbitration tables will be set on the appropriate targets<br><br>
3.2. PathRecord query handling:<br>
OpenSM should be able to enforce the provided policy on client
request.<br>
The overall flow for such requests is: first the request is matched
against the <br>
defined match rules such that the target QoS-Level definition is found.
Given <br>
the QoS-Level a path(s) search is performed with the given restrictions
imposed <br>
by that level. The following two sections describe these steps.<br><br>
One issue not standardized by the IBTA is how Service-ID is carried in
the <br>
PathRecord and MultiPathRecord attributes. There are basically two
options:<br>
a.<x-tab> </x-tab>Replace the SM-Key
field by the Service-ID. In that case no component mask<br>
bit will be assigned to it. Such that if the field is zero
we should treat it <br>
as if the component mask bit is clear. <br>
b. Encode it into spare fields. For PathRecord the first two fields are
reserved <br>
and are 64 bit when combined. The first component mask bit
maps to the first <br>
reserved field and should be used for Service-ID masking.
For MultiPathRecord <br>
attribute there are no adjacent reserve fields that makes a
64 bit field. So <br>
the reserve field following the packet-lifetime (8 bits)
combined with the <br>
reserved field DGIDCount (56 bits) can make the Service-ID.
In this case also <br>
the first reserve field component mask bit should be used as
the Service-ID <br>
component mask bit.<br><br>
<br><br>
3.2.1. Matching rule search:<br>
A rule is "matching" a PathRecord request using the following
criteria:<br>
* Matching rules provide values in a list of either single value, or
range of <br>
values. A PathRecord field is "matching" the rule field
if it is explicitly <br>
noted in the list of values or is one of the values covered by a
range <br>
included in the field values list.<br>
* Only PathRecord fields that have their component mask bit set should
be<br>
compared. <br>
* For a rule to be "matching" a PathRecord request all the rule
fields should be <br>
"matching" their PathRecord fields. Such that a
PathRecord request that does <br>
not have a component mask field set for one of the rule defined
fields can <br>
not match that rule.<br>
* A PathRecord request that have a component mask bit set for one of the
fields <br>
that is not defined by the rule can match the rule.
<br><br>
The algorithm to be used for searching for a rule match might be as
simple as a <br>
sequential search through all rules or enhanced for better performance.
The <br>
semantics of every rule field and its matching PathRecord field are
described <br>
below:<br>
* Source: the SGID or SLID should be part of this group<br>
* Destination: the DGID or DLID should be part of this group<br>
* Service-ID: check if the requested Service-ID (available in the
PathRecord old <br>
SM-Key field) is matching any of this rule Service-IDs<br>
* TClass: check if the PathRecord TClass field is matching<br><br>
3.2.2 PathRecord response generation:<br>
The QoS-Level pointed by the first rule that matches the PathRecord
request <br>
should be used for obtaining the response SL, MTU-Limit, RATE-Limit,
Path-Bits <br>
and TClass. A default QoS-Level should be used if no rule is matching the
query.<br><br>
The efficient algorithm for finding paths that meet the QoS-Level
criteria is <br>
beyond the scope of this RFC and left for the implementer to provide.
However <br>
the criteria by which the paths match the QoS-Level are described
below:<br><br>
* SL: The paths found should all use the given SL. For that sake
PathRecord <br>
algorithm should traverse the path from source to destination only
through <br>
ports that carry a valid VL (not VL15) by the SL2VL map (should
consider input <br>
and output ports and SL). <br>
* MTU-Limit: The resulting paths MTU should not exceed the given
MTU-Limit<br>
* Rate-Limit: The resulting paths RATE should not exceed the given
RATE-Limit <br>
(rate limit is given in units of link BW = Width*Speed according
to IBTA <br>
Specification Vol-1 table-205 p-901 l-24).<br>
* Path-Bits: define the target LID lowest bits (number of bits defined by
the <br>
target port PortInfo.LMC field). The path should traverse the LFT
using the <br>
target port LID with the path-bits set.<br>
* TClass: should be returned in the result PathRecord. When routing is
going to <br>
be supported by OpenSM we might use this field in selecting the
target <br>
router too in a TBD way.<br><br>
_______________________________________________<br>
openib-general mailing list<br>
openib-general@openib.org<br>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a>
</font></blockquote></body>
</html>