[openib-general] question on QoS support
Wang, Feiyi
fwang2 at ornl.gov
Mon Nov 6 10:13:14 PST 2006
Hal -
Please see the output for active port 1 (although there are two ports on
this HCA, the second one is disabled now).
#smpquery portinfo 8 1
# Port info: Lid 8 port 1
Mkey:............................0x0000000000000000
GidPrefix:.......................0xfe80000000000000
Lid:.............................0x0008
SMLid:...........................0x0001
CapMask:.........................0x2510a68
IsTrapSupported
IsAutomaticMigrationSupported
IsSLMappingSupported
IsLedInfoSupported
IsSystemImageGUIDsupported
IsCommunicatonManagementSupported
IsVendorClassSupported
IsCapabilityMaskNoticeSupported
IsClientRegistrationSupported
DiagCode:........................0x0000
MkeyLeasePeriod:.................0
LocalPort:.......................1
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 or 5.0 Gbps
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkDownDefState:................Polling
ProtectBits:.....................0
LMC:.............................0
LinkSpeedActive:.................2.5 Gbps
LinkSpeedEnabled:................2.5 or 5.0 Gbps
NeighborMTU:.....................2048
SMSL:............................0
VLCap:...........................VL0-7
InitType:........................0x00
VLHighLimit:.....................255
VLArbHighCap:....................8
VLArbLowCap:.....................8
InitReply:.......................0x00
MtuCap:..........................2048
VLStallCount:....................7
HoqLife:.........................31
OperVLs:.........................VL0-7
PartEnforceInb:..................0
PartEnforceOutb:.................0
FilterRawInb:....................0
FilterRawOutb:...................0
MkeyViolations:..................0
PkeyViolations:..................0
QkeyViolations:..................0
GuidCap:.........................32
ClientReregister:................0
SubnetTimeout:...................18
RespTimeVal:.....................16
LocalPhysErr:....................8
OverrunErr:......................8
MaxCreditHint:...................0
RoundTrip:.......................0
Feiyi
-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com]
Sent: Friday, November 03, 2006 3:58 PM
To: Wang, Feiyi
Cc: openib-general at openib.org
Subject: RE: [openib-general] question on QoS support
On Fri, 2006-11-03 at 15:56, Wang, Feiyi wrote:
> 255
>
> I think I tested with default 0 before, that is send at most one
packet
> before give low priority table the chance according to IBA. It doesn't
> seem to make a difference though.
I was hoping you would say 0 as that means 1 packet before looking at
low priority.
255 means unbounded packets on high priority. Can you send me the
results of smpquery portinfo on that port to ensure that it is being set
properly ?
-- Hal
> Feiyi
>
>
> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Friday, November 03, 2006 3:51 PM
> To: Wang, Feiyi
> Cc: openib-general at openib.org
> Subject: RE: [openib-general] question on QoS support
>
> On Fri, 2006-11-03 at 15:43, Wang, Feiyi wrote:
> > The test is done on two hosts, say A and B. A has 4x SDR (run
> ib_rdam_bw
> > as server), B has 4x DDR (run more than one thread of ib_rdma_bw as
> > clients). The sl2vl table read as:
> >
> > smpquery sl2vl 7
> > # SL2VL table: Lid 7
> > # SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8|
> 9|10|11|12|13|14|15|
> > ports: in 0, out 0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6|
> 7|
> >
> > smpquery vlarb 7
> > # VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8
> > # Low priority VL Arbitration Table:
> > VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
> > # High priority VL Arbitration Table:
> > VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 |
> >
> > Low priority table entries are all zero to skip.
> > High priority table give VL 0 and VL 2 different weight.
> >
> > The SL is specified on command line, one thread with SL 0, the other
> > thread with SL 2.
> >
> > Thanks for looking into this, and let me know if more info is
needed.
>
> What's the limit of high priority ?
>
> -- Hal
>
> > Feiyi
> >
> >
> >
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > Sent: Friday, November 03, 2006 3:27 PM
> > To: Wang, Feiyi
> > Cc: openib-general at openib.org
> > Subject: Re: [openib-general] question on QoS support
> >
> > On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote:
> > > In our test at the ORNL - it appears you can "turn off" the
traffic
> by
> > > giving every VL weight 0.
> >
> > A weight of 0 indicates to skip that entry.
> >
> > > As soon as you assign non-zero VL weight,
> > > the traffic starts to flow, however, VL with more weight doesn't
> have
> > > expected preference treatment. In other words, traffic shaping
> didn't
> > > take place. smpquery vlarb verified the mapping table was there.
> >
> > correctly ?
> >
> > Is it high or low priority or both ?
> >
> > What about SL2VLMapping table ? Is it setup correctly ?
> >
> > What's your topology for this ?
> >
> > Can you send your SL2VLMapping and VLarbitration configuration ?
> >
> > > I believe the scenario described below 'should' be able to
generate
> > > congestion point ... but it would be helpful if someone can
> elaborate
> > > a way to "look into" how/if scheduling/arbitration take place.
> >
> > The only ways I know would be to look at either the packets on the
> wire
> > or what you are doing with multiple streams which seems valid to me.
> >
> > Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to
> > understand how to configure this ?
> >
> > -- Hal
> >
> > > Best,
> > >
> > > Feiyi
> > >
> > >
> > > On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <halr at voltaire.com>
> > wrote:
> > > > Hi Oliver,
> > > >
> > > > On Thu, 2006-11-02 at 10:20, Oliver wrote:
> > > > > Hi, Hal -
> > > > >
> > > > > > How is this being observed/measured ?
> > > > >
> > > > > Host A, B, with 4x DDR both connected to Flextronic switch.
> > > > > A single process of ibv_read_bw gives about 1415MB /s average
> > > > > bandwidth. Two concurrent process report 714.45 MB/s each,
dead
> > even.
> > > > > Now if I bump up one process with a different SL, then I
expect
> to
> > see
> > > > > shaping to take place. Please let me if the scenario makes
> sense.
> > > >
> > > > It makes sense. However, if the higher priority traffic does not
> > fill
> > > > the scheduling, the low priority can take up the slack so I'm
not
> > sure
> > > > if this is what you are seeing or something else.
> > > >
> > > > It might be interesting to try the same thing at SDR speeds.
> > > >
> > > > -- Hal
> > > >
> > > > > > Yes, 8 VLs should be supported in your subnet. You can
verify
> > this with
> > > > > > smpquery portinfo on the HCA port and examine OperVLs
assuming
> > the port
> > > > > > is ACTIVE.
> > > > >
> > > > > yes, I verified the data VL support, it is 8. I will poke for
> more
> > > > > info with suggested commands by Sasha.
> > > > >
> > > > > > > A related question is, if I modify qos setting in SM, do I
> > need to
> > > > > > > restart SA on each hosts for it to see the changes? (I am
> > hoping not,
> > > > > > > as I tried in the test, it doesn't seem to make a
> difference)
> > > > > >
> > > > > > Not sure what you mean. SA is tightly coupled with the
OpenSM.
> > Do you
> > > > > > mean SA client ? The client hosts don't need restarting but
> did
> > you
> > > > > > restart OpenSM with your QoS configuration ?
> > > > >
> > > > > I mean client SA. yes, I understand OpenSM needs to be
> restarted.
> > > > >
> > > > > > BTW, which OpenSM are you running ?
> > > > >
> > > > > OFED 1.1 based.
> > > > >
> > > > > thanks
> > > > >
> > > > > - Oliver
> > > >
> > > >
> >
>
More information about the general
mailing list