[openib-general] RE: [PATCH 0/2] opensm: low-level QoS implementation

[Eitan Zahavi]

Hi Sasha, Hal,

It is great that you work on QoS implementation. In general I see this
simple extension of the SM capabilities as very useful one. But I think
it would have been better if you first send out the RFC for the proposed
functionality and only later implement it (as was done on the partition
manager case). I have extracted the "description section" from the patch
and here are my comments (prefixed [EZ]) to it:

osm/doc/qos-config.txt:
Trivial low level QoS configuration proposition.
===============================================

Basically we have set of QoS related low-level configuration parameters.
[EZ] I expected QoS parameters to be stored in a QoS Policy file as was
done in
         the partition case. The main reason for that is that I believe
a simple set of 
         parameters is going to be an over simplification of the
required functionality.

All those parameter names are prefixed by "qos_" string. There is full
list of such parameters:

  qos_max_vls    - The number of maximum VLs will be on the Subnet
  qos_high_limit - The limit of High Priority component of VL
Arbitration
                   table (IBA 7.6.9)
  qos_vlarb_low  - High priority VL Arbitration table (IBA 7.6.9)
template.
  qos_vlarb_high - Low priority VL Arbitration table (IBA 7.6.9)
template.
                   Both VL arbitration templates are pairs of VL and
weight.
  qos_sl2vl      - SL2VL Mapping table (IBA 7.6.6) template. It is a
list
                   of VLs corresponding to SLs 0-15. (Note the VL15 used
                   here means drop this SL).

Typical default values (hard-coded in OpenSM initialization) are:

  qos_max_vls=15
  qos_high_limit=0
  qos_vlarb_low=0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0
  qos_vlarb_high=0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4
  qos_sl2vl=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

The syntax is compatible with rest of OpenSM configuration options and
values may be stored in OpenSM config file (cached options file).

[EZ] The above set of parameters is fine for "default" QoS support.
         I better understand the scope of this proposal now.
[EZ] Please note that algorithm to validate the applicability of the
above on the 
         particular fabric is still required as not all devices support
the 16 VLs and not all 
         devices must support VLArb of 8 entries. In such cases we
should at least provide 
          an error describing why the provided setting is un-realizable.
[EZ] The default SL2VL map and VLArb tables are not consistent: The
VLArb tables do 
         not provide any entry for VL > 7 so the SL >= 8 are not usable.
         
In addition to above we may to define separate QoS configuration
parameters sets for various target types. As targets we currently
support
HCA, routers, switch external ports and switch's enhanced port 0. The
names of such specialized parameters are prefixed by "qos_<type>_"
string. There is full list of currently supported sets:

  qos_hca_ - QoS configuration parameters set for HCAs.
  qos_rtr_ - parameters set for routers.
  qos_sw0_ - parameters set for switches' port 0.
  qos_swe_ - parameters set for switches' external ports.

[EZ] I do not see how the above could be used. Instead I do see groups
of nodes as being 
         assigned different QoS levels. As we defined "groups of nodes"
in the partition 
         policy I would propose using the partitions as the means to
define node groups.
[EZ] So I propose to keep the "trivial" implementation without this
level of control. 
         Instead I would prefer having QoS Policy file defined such that
these groups can be 
         referred to.

Examples:

  qos_sw0_max_vls=2
  qos_hca_sl2vl=0,1,2,3,5,5,5,12,12,0,
  qos_swe_high_limit=0

[EZ] Another concept that is not represented in this proposal is the
support of selecting QoS level for particular PathRecord queries. (or
how does the ULP or Application obtain the SL). But I guess this falls
under the second phase of the QoS support.

I will follow-up with proposal for OSM QoS policy file syntax and
functionality RFC for the next implementation steps.

Thanks


Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Sasha Khapyorsky [mailto:sashak at voltaire.com]
> Sent: Monday, May 08, 2006 11:00 PM
> To: Hal Rosenstock; openib-general at openib.org
> Cc: Eitan Zahavi; Yael Kalka; Ofer Gigi; Eli Dorfman
> Subject: [PATCH 0/2] opensm: low-level QoS implementation
> 
> Hello,
> 
> There is support for low level Quality of Service (QoS) parameters
> configuration and setup in OpenSM.
> 
> Please comment. Thanks.
> 
> Sasha.