[ofa-general] RE: QoS for iSER
Kanevsky, Arkady
Arkady.Kanevsky at netapp.com
Wed Nov 14 15:18:08 PST 2007
Yevgeny,
Lets say I want to run another ULP, say NFS-RDMA.
Does this mean by default I will get the SL which is in use by
currently running ULP, say SDP?
Or the SL is differentiated based on port?
And if I want to run different SL level traffic between the same pair
of nodes I will need to use different port ID?
Look strange for ULPs that have well known port ID...
Thanks,
Arkady Kanevsky email: arkady at netapp.com
Network Appliance Inc. phone: 781-768-5395
1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
Waltham, MA 02451 central phone: 781-768-5300
> -----Original Message-----
> From: Yevgeny Kliteynik [mailto:kliteyn at dev.mellanox.co.il]
> Sent: Wednesday, November 14, 2007 3:21 PM
> To: Kanevsky, Arkady
> Cc: Hal Rosenstock; general at lists.openfabrics.org
> Subject: Re: [ofa-general] RE: QoS for iSER
>
> Kanevsky, Arkady wrote:
> > what happens when multiple apps runs on the same server?
>
> I guess that when you say "server" you mean "host" and not
> the server from server-client terminology.
>
> This is what the whole point of QoS is: if the applications
> are using the same ULP, they probably would get the same
> Service Level, unless they were differentiated by the
> administrator is some other way, e.g. they all use SDP, but
> connect to different TCP port of the server application.
> If the applications are using different ULPs, they will get
> Service Level accordingly to the ULPs that they are using.
>
> -- Yevgeny
>
> > Arkady Kanevsky email: arkady at netapp.com
> > Network Appliance Inc. phone: 781-768-5395
> > 1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
> > Waltham, MA 02451 central phone: 781-768-5300
> >
> >
> >> -----Original Message-----
> >> From: Hal Rosenstock [mailto:hrosenstock at xsigo.com]
> >> Sent: Wednesday, November 14, 2007 8:18 AM
> >> To: Yevgeny Kliteynik
> >> Cc: gdror at mellanox.co.il; general at lists.openfabrics.org
> >> Subject: Re: [ofa-general] RE: QoS for iSER
> >>
> >> On Wed, 2007-11-14 at 11:11 +0200, Yevgeny Kliteynik wrote:
> >>> Hal Rosenstock wrote:
> >>>> Or,
> >>>>
> >>>> On 11/13/07, Or Gerlitz <ogerlitz at voltaire.com> wrote:
> >>>>> Yevgeny,
> >>>>>
> >>>>> iSER (as you can learn from doing a grep) is using the
> >> RDMA-CM TCP
> >>>>> port space as does RDS. The RDMA-CM signature is
> >> something which I
> >>>>> am sure exists, you can look on the RDMA-CM IB spec
> >> Annex to see if
> >>>>> such thing indeed exist or I am wrong.
> >>>> Did you really look at the annex for this ?
> >>>>
> >>>>> The TCP port is the 16 bit port portion of the ip:port address
> >>>>> provided by a ULP that uses the RDMA-CM to rdma_resolve_addr(),
> >>>>> again the annex explained how the port is embedded into
> >> the SID, I
> >>>>> don't remember the location within the 64 bit string.
> >>>> It's in the low 16 bits (bytes 6-7) of the SID as the
> >> annex indicates.
> >>>>> Or.
> >>>>>
> >>>>> -------- Original Message --------
> >>>>> Subject:
> >>>>> Re: QoS for iSER
> >>>>> Date:
> >>>>> Mon, 12 Nov 2007 11:41:43 +0200
> >>>>> From: Yevgeny Kliteynik <kliteyn at dev.mellanox.co.il>
> >>>>>
> >>>>> Hi Erez,
> >>>>>
> >>>>> Erez Zilber wrote:
> >>>>>> to create the SID, the rdma cm combines
> >>>>>>
> >>>>>> 1) the port space
> >>>>> What is the port space for iSER?
> >>>>> For SDP it's 0x10000 - 0x1FFFF.
> >>>>> For RDS it's 0x1060000 - 0x106FFFF
> >> I presume this is just saying RDS uses IP protocol TCP and
> there is
> >> no well known port (e.g. uses dynamic ports). So how do you know
> >> ahead of time which port ?
> >>
> >>>>> For iSER it's ...?
> >>>> These numbers are too large for just "port space".
> >>>>
> >>>> iSER SID is 0x000000000106035c
> >>>>
> >>>> in your nomenclature, I guess 0x106035c
> >>>>
> >>>> 01 says RDMA aware ULP service ID range
> >>>> 06 says IP protocol is TCP
> >>>> 0x035c (port 860) is the well known TCP port for iSCSI
> >>> Thanks, that is just what I needed.
> >>> I'm preparing a (very) simplified interface for defining
> QoS policy.
> >>> I'm adding an additional section in QoS policy file,
> where an admin
> >>> will be able to configure QoS per ULP or per application
> w/o going
> >>> into too many details.
> >>> Here's the example of what I have in mind:
> >>>
> >>> qos-ulps
> >>> default : 0 #default SL
> >>> sdp, port 10000-20000 : 2
> >>> sdp : 0 #default SL for SDP
> >>> rds, port 25000 : 2 #SL for RDS when
> >> destination port is 25000
> >>
> >> Isn't there a chicken and egg problem here with this ? How do you
> >> know port 25000 will be assigned "in advance" ?
> >>
> >>> rds, : 0 #default SL for RDS
> >> I don't see how RDS can work separate from other CMA based
> protocols
> >> which use dynamic ports.
> >>
> >>> iser *??????* : 4 #SL for iSER
> >>> ipoib, pkey 0x0001 : 5 #SL for IPoIB on
> >> partition with pkey 0x0001
> >>> ipoib : 6 #default IPoIB
> >> partition - pkey=0x7FFF
> >> ...
> >>> end-qos-ulps
> >>>
> >>> This syntax is possible only if there are well known facts
> >> such as SDP
> >>> service ID, in which case admin will be able to just state "sdp:
> >>> <sl>", and OpenSM will (internally) generate relevant
> matching rule
> >>> and QoS level based on this known service ID.
> >>>
> >>> So back to iSER:
> >>>
> >>> Can I assume that the target port for iSER will always be
> >> 860, hence
> >>> the iSER service ID will always be 0x000000000106035c?
> >> In terms of iSER, I was only commenting on what the spec
> says. I did
> >> not verify its operation in terms of the code.
> >> Does the code follow the spec ?
> >>
> >> -- Hal
> >>
> >>> Or perhaps I can do it similar to SDP, where there is an
> option to
> >>> specify the port ranges along with the ULP name (SDP):
> >>> - if administrator only specifies "iser", I can assume that
> >>> the service ID is default 0x000000000106035c
> >>> - if administrator only specifies "iser" and ports, OpenSM
> >>> will build service ID based on a well known prefix
> >>> (0x000000000106pppp) where the last 4 hex digits are target
> >>> port number
> >>>
> >>> Keep in mind that if this doesn't look too flexible and
> >> doesn't cover
> >>> all the cases, there's always the rest of the QoS policy
> >> file with all
> >>> the advanced configuration.
> >>>
> >>> -- Yevgeny
> >>>
> >>>> -- Hal
> >>>>
> >>>>>> 2) the rdma cm signature
> >>>>> Do you mean something iSER-specific, or just the way the
> >> cm builds
> >>>>> the service ID out of port space and tcp port?
> >>>>> Can you give an example?
> >>>>>
> >>>>>> 3) the destination tcp port provided to rdma_resolve_addr
> >>>>> I guess that tcp port is in the lower 4 nibs of the service ID,
> >>>>> similar to SDP. Right?
> >>>>> -- Yevgeny
> >>>>>
> >>>>> _______________________________________________
> >>>>> general mailing list
> >>>>> general at lists.openfabrics.org
> >>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>>>>
> >>>>> To unsubscribe, please visit
> >>>>> http://openib.org/mailman/listinfo/openib-general
> >>>>>
> >>> _______________________________________________
> >>> general mailing list
> >>> general at lists.openfabrics.org
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>>
> >>> To unsubscribe, please visit
> >>> http://openib.org/mailman/listinfo/openib-general
> >> _______________________________________________
> >> general mailing list
> >> general at lists.openfabrics.org
> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>
> >> To unsubscribe, please visit
> >> http://openib.org/mailman/listinfo/openib-general
> >>
> >
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
More information about the general
mailing list