[ofa-general] RE: QoS for iSER
Yevgeny Kliteynik
kliteyn at dev.mellanox.co.il
Thu Nov 15 00:21:10 PST 2007
Kanevsky, Arkady wrote:
> Yevgeny,
> Lets say I want to run another ULP, say NFS-RDMA.
> Does this mean by default I will get the SL which is in use by
> currently running ULP, say SDP?
> Or the SL is differentiated based on port?
> And if I want to run different SL level traffic between the same pair
> of nodes I will need to use different port ID?
> Look strange for ULPs that have well known port ID...
If you run the same application between the same pair of
nodes, you will get the same SL.
If you run different applications over the same ULP that use
different TCP ports, but you have no way knowing what are those
the TCP ports, you will get the same SL.
If you run different applications over different ULPs that are
using same port space (which produces same service ID range),
and you have no way knowing what are the the TCP ports,
you will get the same SL.
In any other case there is a way to differentiate the traffic.
Connection requests (doesn't matter whether it's ULP or not)
can be differentiated by source guid, destination guid, pkey,
service ID, QoS class, and any combination of all of the above.
Please see the QoS RFC:
http://lists.openfabrics.org/pipermail/general/2007-July/038488.html
Check the QoS Policy File in the RFC. There were many changes since
this RFC was issued, but it's still enough to get the general idea.
-- Yevgeny
> Thanks,
>
> Arkady Kanevsky email: arkady at netapp.com
> Network Appliance Inc. phone: 781-768-5395
> 1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
> Waltham, MA 02451 central phone: 781-768-5300
>
>
>> -----Original Message-----
>> From: Yevgeny Kliteynik [mailto:kliteyn at dev.mellanox.co.il]
>> Sent: Wednesday, November 14, 2007 3:21 PM
>> To: Kanevsky, Arkady
>> Cc: Hal Rosenstock; general at lists.openfabrics.org
>> Subject: Re: [ofa-general] RE: QoS for iSER
>>
>> Kanevsky, Arkady wrote:
>>> what happens when multiple apps runs on the same server?
>> I guess that when you say "server" you mean "host" and not
>> the server from server-client terminology.
>>
>> This is what the whole point of QoS is: if the applications
>> are using the same ULP, they probably would get the same
>> Service Level, unless they were differentiated by the
>> administrator is some other way, e.g. they all use SDP, but
>> connect to different TCP port of the server application.
>> If the applications are using different ULPs, they will get
>> Service Level accordingly to the ULPs that they are using.
>>
>> -- Yevgeny
>>
>>> Arkady Kanevsky email: arkady at netapp.com
>>> Network Appliance Inc. phone: 781-768-5395
>>> 1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
>>> Waltham, MA 02451 central phone: 781-768-5300
>>>
>>>
>>>> -----Original Message-----
>>>> From: Hal Rosenstock [mailto:hrosenstock at xsigo.com]
>>>> Sent: Wednesday, November 14, 2007 8:18 AM
>>>> To: Yevgeny Kliteynik
>>>> Cc: gdror at mellanox.co.il; general at lists.openfabrics.org
>>>> Subject: Re: [ofa-general] RE: QoS for iSER
>>>>
>>>> On Wed, 2007-11-14 at 11:11 +0200, Yevgeny Kliteynik wrote:
>>>>> Hal Rosenstock wrote:
>>>>>> Or,
>>>>>>
>>>>>> On 11/13/07, Or Gerlitz <ogerlitz at voltaire.com> wrote:
>>>>>>> Yevgeny,
>>>>>>>
>>>>>>> iSER (as you can learn from doing a grep) is using the
>>>> RDMA-CM TCP
>>>>>>> port space as does RDS. The RDMA-CM signature is
>>>> something which I
>>>>>>> am sure exists, you can look on the RDMA-CM IB spec
>>>> Annex to see if
>>>>>>> such thing indeed exist or I am wrong.
>>>>>> Did you really look at the annex for this ?
>>>>>>
>>>>>>> The TCP port is the 16 bit port portion of the ip:port address
>>>>>>> provided by a ULP that uses the RDMA-CM to rdma_resolve_addr(),
>>>>>>> again the annex explained how the port is embedded into
>>>> the SID, I
>>>>>>> don't remember the location within the 64 bit string.
>>>>>> It's in the low 16 bits (bytes 6-7) of the SID as the
>>>> annex indicates.
>>>>>>> Or.
>>>>>>>
>>>>>>> -------- Original Message --------
>>>>>>> Subject:
>>>>>>> Re: QoS for iSER
>>>>>>> Date:
>>>>>>> Mon, 12 Nov 2007 11:41:43 +0200
>>>>>>> From: Yevgeny Kliteynik <kliteyn at dev.mellanox.co.il>
>>>>>>>
>>>>>>> Hi Erez,
>>>>>>>
>>>>>>> Erez Zilber wrote:
>>>>>>>> to create the SID, the rdma cm combines
>>>>>>>>
>>>>>>>> 1) the port space
>>>>>>> What is the port space for iSER?
>>>>>>> For SDP it's 0x10000 - 0x1FFFF.
>>>>>>> For RDS it's 0x1060000 - 0x106FFFF
>>>> I presume this is just saying RDS uses IP protocol TCP and
>> there is
>>>> no well known port (e.g. uses dynamic ports). So how do you know
>>>> ahead of time which port ?
>>>>
>>>>>>> For iSER it's ...?
>>>>>> These numbers are too large for just "port space".
>>>>>>
>>>>>> iSER SID is 0x000000000106035c
>>>>>>
>>>>>> in your nomenclature, I guess 0x106035c
>>>>>>
>>>>>> 01 says RDMA aware ULP service ID range
>>>>>> 06 says IP protocol is TCP
>>>>>> 0x035c (port 860) is the well known TCP port for iSCSI
>>>>> Thanks, that is just what I needed.
>>>>> I'm preparing a (very) simplified interface for defining
>> QoS policy.
>>>>> I'm adding an additional section in QoS policy file,
>> where an admin
>>>>> will be able to configure QoS per ULP or per application
>> w/o going
>>>>> into too many details.
>>>>> Here's the example of what I have in mind:
>>>>>
>>>>> qos-ulps
>>>>> default : 0 #default SL
>>>>> sdp, port 10000-20000 : 2
>>>>> sdp : 0 #default SL for SDP
>>>>> rds, port 25000 : 2 #SL for RDS when
>>>> destination port is 25000
>>>>
>>>> Isn't there a chicken and egg problem here with this ? How do you
>>>> know port 25000 will be assigned "in advance" ?
>>>>
>>>>> rds, : 0 #default SL for RDS
>>>> I don't see how RDS can work separate from other CMA based
>> protocols
>>>> which use dynamic ports.
>>>>
>>>>> iser *??????* : 4 #SL for iSER
>>>>> ipoib, pkey 0x0001 : 5 #SL for IPoIB on
>>>> partition with pkey 0x0001
>>>>> ipoib : 6 #default IPoIB
>>>> partition - pkey=0x7FFF
>>>> ...
>>>>> end-qos-ulps
>>>>>
>>>>> This syntax is possible only if there are well known facts
>>>> such as SDP
>>>>> service ID, in which case admin will be able to just state "sdp:
>>>>> <sl>", and OpenSM will (internally) generate relevant
>> matching rule
>>>>> and QoS level based on this known service ID.
>>>>>
>>>>> So back to iSER:
>>>>>
>>>>> Can I assume that the target port for iSER will always be
>>>> 860, hence
>>>>> the iSER service ID will always be 0x000000000106035c?
>>>> In terms of iSER, I was only commenting on what the spec
>> says. I did
>>>> not verify its operation in terms of the code.
>>>> Does the code follow the spec ?
>>>>
>>>> -- Hal
>>>>
>>>>> Or perhaps I can do it similar to SDP, where there is an
>> option to
>>>>> specify the port ranges along with the ULP name (SDP):
>>>>> - if administrator only specifies "iser", I can assume that
>>>>> the service ID is default 0x000000000106035c
>>>>> - if administrator only specifies "iser" and ports, OpenSM
>>>>> will build service ID based on a well known prefix
>>>>> (0x000000000106pppp) where the last 4 hex digits are target
>>>>> port number
>>>>>
>>>>> Keep in mind that if this doesn't look too flexible and
>>>> doesn't cover
>>>>> all the cases, there's always the rest of the QoS policy
>>>> file with all
>>>>> the advanced configuration.
>>>>>
>>>>> -- Yevgeny
>>>>>
>>>>>> -- Hal
>>>>>>
>>>>>>>> 2) the rdma cm signature
>>>>>>> Do you mean something iSER-specific, or just the way the
>>>> cm builds
>>>>>>> the service ID out of port space and tcp port?
>>>>>>> Can you give an example?
>>>>>>>
>>>>>>>> 3) the destination tcp port provided to rdma_resolve_addr
>>>>>>> I guess that tcp port is in the lower 4 nibs of the service ID,
>>>>>>> similar to SDP. Right?
>>>>>>> -- Yevgeny
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> general mailing list
>>>>>>> general at lists.openfabrics.org
>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>>
>>>>>>> To unsubscribe, please visit
>>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>>
>>>>> _______________________________________________
>>>>> general mailing list
>>>>> general at lists.openfabrics.org
>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>
>>>>> To unsubscribe, please visit
>>>>> http://openib.org/mailman/listinfo/openib-general
>>>> _______________________________________________
>>>> general mailing list
>>>> general at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>
>>>> To unsubscribe, please visit
>>>> http://openib.org/mailman/listinfo/openib-general
>>>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
>
More information about the general
mailing list