[openib-general] question on QoS support

Fri Nov 3 12:57:43 PST 2006

On Fri, 2006-11-03 at 15:56, Wang, Feiyi wrote:
> 255 
> 
> I think I tested with default 0 before, that is send at most one packet
> before give low priority table the chance according to IBA. It doesn't
> seem to make a difference though.

I was hoping you would say 0 as that means 1 packet before looking at
low priority.

255 means unbounded packets on high priority. Can you send me the
results of smpquery portinfo on that port to ensure that it is being set
properly ?

-- Hal 

> Feiyi
> 
> 
> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com] 
> Sent: Friday, November 03, 2006 3:51 PM
> To: Wang, Feiyi
> Cc: openib-general at openib.org
> Subject: RE: [openib-general] question on QoS support
> 
> On Fri, 2006-11-03 at 15:43, Wang, Feiyi wrote:
> > The test is done on two hosts, say A and B. A has 4x SDR (run
> ib_rdam_bw
> > as server), B has 4x DDR (run more than one thread of ib_rdma_bw as
> > clients). The sl2vl table read as:
> > 
> > smpquery sl2vl 7
> > # SL2VL table: Lid 7
> > #                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8|
> 9|10|11|12|13|14|15|
> > ports: in  0, out  0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6|
> 7|
> > 
> > smpquery vlarb  7
> > # VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8
> > # Low priority VL Arbitration Table:
> > VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
> > # High priority VL Arbitration Table:
> > VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> > WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 |
> > 
> > Low priority table entries are all zero to skip.
> > High priority table give VL 0 and VL 2 different weight.
> > 
> > The SL is specified on command line, one thread with SL 0, the other
> > thread with SL 2.
> > 
> > Thanks for looking into this, and let me know if more info is needed.
> 
> What's the limit of high priority ?
> 
> -- Hal
> 
> > Feiyi
> > 
> > 
> > 
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com] 
> > Sent: Friday, November 03, 2006 3:27 PM
> > To: Wang, Feiyi
> > Cc: openib-general at openib.org
> > Subject: Re: [openib-general] question on QoS support
> > 
> > On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote:
> > > In our test at the ORNL - it appears you can "turn off" the traffic
> by
> > > giving every VL weight 0.
> > 
> > A weight of 0 indicates to skip that entry.
> > 
> > >  As soon as you assign non-zero VL weight,
> > > the traffic starts to flow, however, VL with more weight doesn't
> have
> > > expected preference treatment. In other words, traffic shaping
> didn't
> > > take place. smpquery vlarb verified the mapping table was there.
> > 
> > correctly ?
> > 
> > Is it high or low priority or both ?
> > 
> > What about SL2VLMapping table ? Is it setup correctly ?
> > 
> > What's your topology for this ?
> > 
> > Can you send your SL2VLMapping and VLarbitration configuration ?
> > 
> > > I believe the scenario described below 'should' be able to generate
> > > congestion point ... but it would be helpful if someone can
> elaborate
> > > a way to "look into" how/if scheduling/arbitration take place.
> > 
> > The only ways I know would be to look at either the packets on the
> wire
> > or what you are doing with multiple streams which seems valid to me.
> > 
> > Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to
> > understand how to configure this ?
> > 
> > -- Hal
> > 
> > > Best,
> > > 
> > > Feiyi
> > > 
> > > 
> > > On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <halr at voltaire.com>
> > wrote:
> > > > Hi Oliver,
> > > >
> > > > On Thu, 2006-11-02 at 10:20, Oliver wrote:
> > > > > Hi, Hal -
> > > > >
> > > > > > How is this being observed/measured ?
> > > > >
> > > > > Host A, B, with 4x DDR both connected to Flextronic switch.
> > > > > A single process of ibv_read_bw gives about 1415MB /s average
> > > > > bandwidth. Two concurrent process report 714.45 MB/s each, dead
> > even.
> > > > > Now if I bump up one process with a different SL, then I expect
> to
> > see
> > > > > shaping to take place. Please let me if the scenario makes
> sense.
> > > >
> > > > It makes sense. However, if the higher priority traffic does not
> > fill
> > > > the scheduling, the low priority can take up the slack so I'm not
> > sure
> > > > if this is what you are seeing or something else.
> > > >
> > > > It might be interesting to try the same thing at SDR speeds.
> > > >
> > > > -- Hal
> > > >
> > > > > > Yes, 8 VLs should be supported in your subnet. You can verify
> > this with
> > > > > > smpquery portinfo on the HCA port and examine OperVLs assuming
> > the port
> > > > > > is ACTIVE.
> > > > >
> > > > > yes, I verified the data VL support, it is 8. I will poke for
> more
> > > > > info with suggested commands by Sasha.
> > > > >
> > > > > > > A related question is, if I modify qos setting in SM, do I
> > need to
> > > > > > > restart SA on each hosts for it to see the changes? (I am
> > hoping not,
> > > > > > > as I tried in the test, it doesn't seem to make a
> difference)
> > > > > >
> > > > > > Not sure what you mean. SA is tightly coupled with the OpenSM.
> > Do you
> > > > > > mean SA client ? The client hosts don't need restarting but
> did
> > you
> > > > > > restart OpenSM with your QoS configuration ?
> > > > >
> > > > > I mean client SA. yes, I understand OpenSM needs to be
> restarted.
> > > > >
> > > > > > BTW, which OpenSM are you running ?
> > > > >
> > > > > OFED 1.1 based.
> > > > >
> > > > > thanks
> > > > >
> > > > > - Oliver
> > > >
> > > >
> > 
> 