[openib-general] Re: [PATCH 0/2] opensm: low-level QoS implementation

[Sasha Khapyorsky]

Hi Eitan,

On 14:20 Tue 09 May     , Eitan Zahavi wrote:
> 
> It is great that you work on QoS implementation. In general I see this
> simple extension of the SM capabilities as very useful one. But I think
> it would have been better if you first send out the RFC for the proposed
> functionality and only later implement it (as was done on the partition
> manager case).

I agree with you, those patches are RFC (I've just forgot to sign this
in the subject).

> I have extracted the "description section" from the patch
> and here are my comments (prefixed [EZ]) to it:
> 
> osm/doc/qos-config.txt:
> Trivial low level QoS configuration proposition.
> ===============================================
> 
> Basically we have set of QoS related low-level configuration parameters.
> [EZ] I expected QoS parameters to be stored in a QoS Policy file as was
> done in
>          the partition case. The main reason for that is that I believe
> a simple set of 
>          parameters is going to be an over simplification of the
> required functionality.

The main goal of this implementation is to provide low-level primitives
to setup QoS related attributes and simple but useful way to configure
it. In this case just "raw" configuration parameters seems as most
suitable for me. So unlike to partition case we don't really have any
"policy" yet.

I think kind of this will be the next stage of QoS functionality. Some
things we may do right now - for instance we may associate service level
value with particular partition (and define it in partition "policy"
configuration).

> All those parameter names are prefixed by "qos_" string. There is full
> list of such parameters:
> 
>   qos_max_vls    - The number of maximum VLs will be on the Subnet
>   qos_high_limit - The limit of High Priority component of VL
> Arbitration
>                    table (IBA 7.6.9)
>   qos_vlarb_low  - High priority VL Arbitration table (IBA 7.6.9)
> template.
>   qos_vlarb_high - Low priority VL Arbitration table (IBA 7.6.9)
> template.
>                    Both VL arbitration templates are pairs of VL and
> weight.
>   qos_sl2vl      - SL2VL Mapping table (IBA 7.6.6) template. It is a
> list
>                    of VLs corresponding to SLs 0-15. (Note the VL15 used
>                    here means drop this SL).
> 
> Typical default values (hard-coded in OpenSM initialization) are:
> 
>   qos_max_vls=15
>   qos_high_limit=0
>   qos_vlarb_low=0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0
>   qos_vlarb_high=0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4
>   qos_sl2vl=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
> 
> The syntax is compatible with rest of OpenSM configuration options and
> values may be stored in OpenSM config file (cached options file).
> 
> [EZ] The above set of parameters is fine for "default" QoS support.
>          I better understand the scope of this proposal now.
> [EZ] Please note that algorithm to validate the applicability of the
> above on the 
>          particular fabric is still required as not all devices support
> the 16 VLs

VL numbers are translated according to port's capabilities and
configured OperVLs (the numbers are MODed).

> and not all 
>          devices must support VLArb of 8 entries. In such cases we
> should at least provide 
>           an error describing why the provided setting is un-realizable.

In the case of "short" VLArb table the template will be truncated
(silently) to meet port's capabilities. This is not a error, right?

> [EZ] The default SL2VL map and VLArb tables are not consistent: The
> VLArb tables do 
>          not provide any entry for VL > 7 so the SL >= 8 are not usable.

This is reasonable note, we may extend default set.

> In addition to above we may to define separate QoS configuration
> parameters sets for various target types. As targets we currently
> support
> HCA, routers, switch external ports and switch's enhanced port 0. The
> names of such specialized parameters are prefixed by "qos_<type>_"
> string. There is full list of currently supported sets:
> 
>   qos_hca_ - QoS configuration parameters set for HCAs.
>   qos_rtr_ - parameters set for routers.
>   qos_sw0_ - parameters set for switches' port 0.
>   qos_swe_ - parameters set for switches' external ports.
> 
> [EZ] I do not see how the above could be used. Instead I do see groups
> of nodes as being 
>          assigned different QoS levels. As we defined "groups of nodes"
> in the partition 
>          policy I would propose using the partitions as the means to
> define node groups.
> [EZ] So I propose to keep the "trivial" implementation without this
> level of control.

At least it does not hurt, and somebody tell me that this will be
useful. So I would prefer to start with this feature.

>          Instead I would prefer having QoS Policy file defined such that
> these groups can be 
>          referred to.

I think that finally we will end with something like this, but for this
we don't have yet some important things (like QoS level), so I would
prefer to start with simple model.

> Examples:
> 
>   qos_sw0_max_vls=2
>   qos_hca_sl2vl=0,1,2,3,5,5,5,12,12,0,
>   qos_swe_high_limit=0
> 
> [EZ] Another concept that is not represented in this proposal is the
> support of selecting QoS level for particular PathRecord queries.

There is no QoS level, just SL, this is not the same.

> (or
> how does the ULP or Application obtain the SL). But I guess this falls
> under the second phase of the QoS support.

We may do do some elements of this soon - specifically SL value per
partition (optional), when SL value may be returned in SA queries.
Obviously low-level QoS implementation is needed for this.

Thanks for comments.

Sasha.