[openib-general] question on QoS support

Mon Nov 6 14:40:28 PST 2006

On Mon, 2006-11-06 at 13:13, Wang, Feiyi wrote:
> Hal -
> 
> Please see the output for active port 1 (although there are two ports on
> this HCA, the second one is disabled now).
> 
> #smpquery portinfo 8 1
> # Port info: Lid 8 port 1
> Mkey:............................0x0000000000000000
> GidPrefix:.......................0xfe80000000000000
> Lid:.............................0x0008
> SMLid:...........................0x0001
> CapMask:.........................0x2510a68
>                                 IsTrapSupported
>                                 IsAutomaticMigrationSupported
>                                 IsSLMappingSupported
>                                 IsLedInfoSupported
>                                 IsSystemImageGUIDsupported
>                                 IsCommunicatonManagementSupported
>                                 IsVendorClassSupported
>                                 IsCapabilityMaskNoticeSupported
>                                 IsClientRegistrationSupported
> DiagCode:........................0x0000
> MkeyLeasePeriod:.................0
> LocalPort:.......................1
> LinkWidthEnabled:................1X or 4X 
> LinkWidthSupported:..............1X or 4X
> LinkWidthActive:.................4X
> LinkSpeedSupported:..............2.5 or 5.0 Gbps
> LinkState:.......................Active
> PhysLinkState:...................LinkUp
> LinkDownDefState:................Polling
> ProtectBits:.....................0
> LMC:.............................0
> LinkSpeedActive:.................2.5 Gbps
> LinkSpeedEnabled:................2.5 or 5.0 Gbps
> NeighborMTU:.....................2048
> SMSL:............................0
> VLCap:...........................VL0-7
> InitType:........................0x00
> VLHighLimit:.....................255

OK; this is pretty conclusive.

> VLArbHighCap:....................8
> VLArbLowCap:.....................8
> InitReply:.......................0x00
> MtuCap:..........................2048
> VLStallCount:....................7
> HoqLife:.........................31
> OperVLs:.........................VL0-7
> PartEnforceInb:..................0
> PartEnforceOutb:.................0
> FilterRawInb:....................0
> FilterRawOutb:...................0
> MkeyViolations:..................0
> PkeyViolations:..................0
> QkeyViolations:..................0
> GuidCap:.........................32
> ClientReregister:................0
> SubnetTimeout:...................18
> RespTimeVal:.....................16
> LocalPhysErr:....................8
> OverrunErr:......................8
> MaxCreditHint:...................0
> RoundTrip:.......................0

Do you have an IB analyzer ?

-- Hal

> Feiyi
> 
> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com] 
> Sent: Friday, November 03, 2006 3:58 PM
> To: Wang, Feiyi
> Cc: openib-general at openib.org
> Subject: RE: [openib-general] question on QoS support
> 
> On Fri, 2006-11-03 at 15:56, Wang, Feiyi wrote:
> > 255 
> > 
> > I think I tested with default 0 before, that is send at most one
> packet
> > before give low priority table the chance according to IBA. It doesn't
> > seem to make a difference though.
> 
> I was hoping you would say 0 as that means 1 packet before looking at
> low priority.
> 
> 255 means unbounded packets on high priority. Can you send me the
> results of smpquery portinfo on that port to ensure that it is being set
> properly ?
> 
> -- Hal 
> 
> > Feiyi
> > 
> > 
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com] 
> > Sent: Friday, November 03, 2006 3:51 PM
> > To: Wang, Feiyi
> > Cc: openib-general at openib.org
> > Subject: RE: [openib-general] question on QoS support
> > 
> > On Fri, 2006-11-03 at 15:43, Wang, Feiyi wrote:
> > > The test is done on two hosts, say A and B. A has 4x SDR (run
> > ib_rdam_bw
> > > as server), B has 4x DDR (run more than one thread of ib_rdma_bw as
> > > clients). The sl2vl table read as:
> > > 
> > > smpquery sl2vl 7
> > > # SL2VL table: Lid 7
> > > #                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8|
> > 9|10|11|12|13|14|15|
> > > ports: in  0, out  0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6|
> > 7|
> > > 
> > > smpquery vlarb  7
> > > # VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8
> > > # Low priority VL Arbitration Table:
> > > VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > > WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
> > > # High priority VL Arbitration Table:
> > > VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > > WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 |
> > > 
> > > Low priority table entries are all zero to skip.
> > > High priority table give VL 0 and VL 2 different weight.
> > > 
> > > The SL is specified on command line, one thread with SL 0, the other
> > > thread with SL 2.
> > > 
> > > Thanks for looking into this, and let me know if more info is
> needed.
> > 
> > What's the limit of high priority ?
> > 
> > -- Hal
> > 
> > > Feiyi
> > > 
> > > 
> > > 
> > > -----Original Message-----
> > > From: Hal Rosenstock [mailto:halr at voltaire.com] 
> > > Sent: Friday, November 03, 2006 3:27 PM
> > > To: Wang, Feiyi
> > > Cc: openib-general at openib.org
> > > Subject: Re: [openib-general] question on QoS support
> > > 
> > > On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote:
> > > > In our test at the ORNL - it appears you can "turn off" the
> traffic
> > by
> > > > giving every VL weight 0.
> > > 
> > > A weight of 0 indicates to skip that entry.
> > > 
> > > >  As soon as you assign non-zero VL weight,
> > > > the traffic starts to flow, however, VL with more weight doesn't
> > have
> > > > expected preference treatment. In other words, traffic shaping
> > didn't
> > > > take place. smpquery vlarb verified the mapping table was there.
> > > 
> > > correctly ?
> > > 
> > > Is it high or low priority or both ?
> > > 
> > > What about SL2VLMapping table ? Is it setup correctly ?
> > > 
> > > What's your topology for this ?
> > > 
> > > Can you send your SL2VLMapping and VLarbitration configuration ?
> > > 
> > > > I believe the scenario described below 'should' be able to
> generate
> > > > congestion point ... but it would be helpful if someone can
> > elaborate
> > > > a way to "look into" how/if scheduling/arbitration take place.
> > > 
> > > The only ways I know would be to look at either the packets on the
> > wire
> > > or what you are doing with multiple streams which seems valid to me.
> > > 
> > > Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to
> > > understand how to configure this ?
> > > 
> > > -- Hal
> > > 
> > > > Best,
> > > > 
> > > > Feiyi
> > > > 
> > > > 
> > > > On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <halr at voltaire.com>
> > > wrote:
> > > > > Hi Oliver,
> > > > >
> > > > > On Thu, 2006-11-02 at 10:20, Oliver wrote:
> > > > > > Hi, Hal -
> > > > > >
> > > > > > > How is this being observed/measured ?
> > > > > >
> > > > > > Host A, B, with 4x DDR both connected to Flextronic switch.
> > > > > > A single process of ibv_read_bw gives about 1415MB /s average
> > > > > > bandwidth. Two concurrent process report 714.45 MB/s each,
> dead
> > > even.
> > > > > > Now if I bump up one process with a different SL, then I
> expect
> > to
> > > see
> > > > > > shaping to take place. Please let me if the scenario makes
> > sense.
> > > > >
> > > > > It makes sense. However, if the higher priority traffic does not
> > > fill
> > > > > the scheduling, the low priority can take up the slack so I'm
> not
> > > sure
> > > > > if this is what you are seeing or something else.
> > > > >
> > > > > It might be interesting to try the same thing at SDR speeds.
> > > > >
> > > > > -- Hal
> > > > >
> > > > > > > Yes, 8 VLs should be supported in your subnet. You can
> verify
> > > this with
> > > > > > > smpquery portinfo on the HCA port and examine OperVLs
> assuming
> > > the port
> > > > > > > is ACTIVE.
> > > > > >
> > > > > > yes, I verified the data VL support, it is 8. I will poke for
> > more
> > > > > > info with suggested commands by Sasha.
> > > > > >
> > > > > > > > A related question is, if I modify qos setting in SM, do I
> > > need to
> > > > > > > > restart SA on each hosts for it to see the changes? (I am
> > > hoping not,
> > > > > > > > as I tried in the test, it doesn't seem to make a
> > difference)
> > > > > > >
> > > > > > > Not sure what you mean. SA is tightly coupled with the
> OpenSM.
> > > Do you
> > > > > > > mean SA client ? The client hosts don't need restarting but
> > did
> > > you
> > > > > > > restart OpenSM with your QoS configuration ?
> > > > > >
> > > > > > I mean client SA. yes, I understand OpenSM needs to be
> > restarted.
> > > > > >
> > > > > > > BTW, which OpenSM are you running ?
> > > > > >
> > > > > > OFED 1.1 based.
> > > > > >
> > > > > > thanks
> > > > > >
> > > > > > - Oliver
> > > > >
> > > > >
> > > 
> > 
> 