[ofa-general] RE: QoS for iSER

Wed Nov 14 12:14:20 PST 2007

Hal Rosenstock wrote:
> On Wed, 2007-11-14 at 17:30 +0200, Yevgeny Kliteynik wrote:
>> Hal Rosenstock wrote:
>>> On Wed, 2007-11-14 at 11:11 +0200, Yevgeny Kliteynik wrote:
>>>> Hal Rosenstock wrote:
>>>>> Or,
>>>>>
>>>>> On 11/13/07, Or Gerlitz <ogerlitz at voltaire.com> wrote:
>>>>>> Yevgeny,
>>>>>>
>>>>>> iSER (as you can learn from doing a grep) is using the RDMA-CM TCP port
>>>>>> space as does RDS. The RDMA-CM signature is something which I am sure
>>>>>> exists, you can look on the RDMA-CM IB spec Annex to see if such thing
>>>>>> indeed exist or I am wrong.
>>>>> Did you really look at the annex for this ?
>>>>>
>>>>>> The TCP port is the 16 bit port portion of
>>>>>> the
>>>>>> ip:port address provided by a ULP that uses the RDMA-CM to
>>>>>> rdma_resolve_addr(), again the annex explained how the port is embedded
>>>>>> into the SID, I don't remember the location within the 64 bit string.
>>>>> It's in the low 16 bits (bytes 6-7) of the SID as the annex indicates.
>>>>>
>>>>>> Or.
>>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject:
>>>>>> Re: QoS for iSER
>>>>>> Date:
>>>>>> Mon, 12 Nov 2007 11:41:43 +0200
>>>>>> From: Yevgeny Kliteynik <kliteyn at dev.mellanox.co.il>
>>>>>>
>>>>>> Hi Erez,
>>>>>>
>>>>>> Erez Zilber wrote:
>>>>>>> to create the SID, the rdma cm combines
>>>>>>>
>>>>>>> 1)       the port space
>>>>>> What is the port space for iSER?
>>>>>> For SDP it's 0x10000 - 0x1FFFF.
>>>>>> For RDS it's 0x1060000 - 0x106FFFF
>>> I presume this is just saying RDS uses IP protocol TCP and there is no
>>> well known port (e.g. uses dynamic ports). So how do you know ahead of
>>> time which port ?
>> See below.
>>
>>>>>> For iSER it's ...?
>>>>> These numbers are too large for just "port space".
>>>>>
>>>>> iSER SID is 0x000000000106035c
>>>>>
>>>>> in your nomenclature, I guess 0x106035c
>>>>>
>>>>> 01 says RDMA aware ULP service ID range
>>>>> 06 says IP protocol is TCP
>>>>> 0x035c (port 860) is the well known TCP port for iSCSI
>>>> Thanks, that is just what I needed.
>>>> I'm preparing a (very) simplified interface for defining QoS policy.
>>>> I'm adding an additional section in QoS policy file, where an admin
>>>> will be able to configure QoS per ULP or per application w/o going
>>>> into too many details.
>>>> Here's the example of what I have in mind:
>>>>
>>>>    qos-ulps
>>>>        default                       : 0 #default SL
>>>>        sdp, port 10000-20000         : 2
>>>>        sdp                           : 0 #default SL for SDP
>>>>        rds, port 25000               : 2 #SL for RDS when destination port is 25000
>>> Isn't there a chicken and egg problem here with this ? How do you know
>>> port 25000 will be assigned "in advance" ?
>> See below.
>>
>>>>        rds,                          : 0 #default SL for RDS
>>> I don't see how RDS can work separate from other CMA based protocols
>>> which use dynamic ports.
>> You're right, it can't.
>> Moreover, as you know, OpenSM is not aware of the term "protocol"
>> at all - it just sees Service IDs in PathRecord request, so the
>> only differentiation it can create is by assigning certain SL to
>> a certain service ID.
> 
> Yes, I could see that you are trying to use protocol as an identifier to
> simplify configuration for a ServiceID range.
> 
>> What I'm trying to do here is to provide a simple way to configure
>> QoS that will work in *many* cases, not in *all* cases.
>>
>> I'm giving the following options to differentiate traffic:
>>   - All the SDP traffic
>>   - Selected ports of the SDP traffic
>>   - SRP traffic by providing SRT target guids
>>   - iSER traffic (to a well known TCP port)
>>   - All the RDS traffic
>>   - Selected ports of the RDS traffic
>>   - IPoIB by pkey
>>   - Specific Service IDs
>>   - Specific pkeys
>>   - Specific destination port guids
> 
> Understood. 
> 
>> And as you've mentioned, some rules may overlap. For instance,
>> if the rule for all the RDS traffic will appear before the iSER
>> rule, then iSER requests will be caught by the RDS rule.
> 
> That doesn't sound so good but I don't see a good alternative here other
> than for this case to put the iSER rule first.

Right

> The other fallback is the
> more detailed configuration but RDS falls into the generic range
> category which is problematic in terms of this (and can't be
> differentiated by ServiceID unlike the other ULPs).

Right again.
If in addition to RDS there are other ULPs in the fabric that fall
in the same range of service ID, then there's no escape from the
rest of the policy file.
For instance, RDS traffic might be differentiated by TCP port range
(service ID range) and target port guids.
But of course, you might find real life examples where nothing helps -
RDS and some other ULP traffic are using same TCP port range, using
the same port guids as source, and using the same guids as target.
In this case I give up - they would end up using the same SL.

-- Yevgeny

> -- Hal
> 
>> There's alway an option to use the rest of the policy file, and
>> then you can create more complex rules to differentiate applications
>> and protocols (i.e. by using service ID range AND/OR source port
>> groups AND/OR destination port groups AND/OR by QoS class.
>>
>> -- Yevgeny
>>
>>>>        iser    *??????*              : 4 #SL for iSER
>>>>        ipoib, pkey 0x0001            : 5 #SL for IPoIB on partition with pkey 0x0001
>>>>        ipoib                         : 6 #default IPoIB partition - pkey=0x7FFF
>>>        ...
>>>>    end-qos-ulps
>>>>
>>>> This syntax is possible only if there are well known facts
>>>> such as SDP service ID, in which case admin will be able to
>>>> just state "sdp: <sl>", and OpenSM will (internally) generate
>>>> relevant matching rule and QoS level based on this known
>>>> service ID.
>>>>
>>>> So back to iSER:
>>>>
>>>> Can I assume that the target port for iSER will always be 860,
>>>> hence the iSER service ID will always be 0x000000000106035c?
>>> In terms of iSER, I was only commenting on what the spec says. I did not
>>> verify its operation in terms of the code. Does the code follow the
>>> spec ?
>>>
>>> -- Hal
>>>
>>>> Or perhaps I can do it similar to SDP, where there is an option
>>>> to specify the port ranges along with the ULP name (SDP):
>>>>   - if administrator only specifies "iser", I can assume that
>>>>     the service ID is default 0x000000000106035c
>>>>   - if administrator only specifies "iser" and ports, OpenSM
>>>>     will build service ID based on a well known prefix
>>>>     (0x000000000106pppp) where the last 4 hex digits are target
>>>>     port number
>>>>
>>>> Keep in mind that if this doesn't look too flexible and doesn't
>>>> cover all the cases, there's always the rest of the QoS policy
>>>> file with all the advanced configuration.
>>>>
>>>> -- Yevgeny
>>>>
>>>>> -- Hal
>>>>>
>>>>>>> 2)       the rdma cm signature
>>>>>> Do you mean something iSER-specific, or just the way the cm
>>>>>> builds the service ID out of port space and tcp port?
>>>>>> Can you give an example?
>>>>>>
>>>>>>> 3)       the destination tcp port provided to rdma_resolve_addr
>>>>>> I guess that tcp port is in the lower 4 nibs of the service ID,
>>>>>> similar to SDP. Right?
>>>>>> -- Yevgeny
>>>>>>
>>>>>> _______________________________________________
>>>>>> general mailing list
>>>>>> general at lists.openfabrics.org
>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>
>>>>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>> _______________________________________________
>>>> general mailing list
>>>> general at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>
>>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 