[openib-general] question on QoS support

[Hal Rosenstock]

On Fri, 2006-11-03 at 15:43, Wang, Feiyi wrote:
> The test is done on two hosts, say A and B. A has 4x SDR (run ib_rdam_bw
> as server), B has 4x DDR (run more than one thread of ib_rdma_bw as
> clients). The sl2vl table read as:
> 
> smpquery sl2vl 7
> # SL2VL table: Lid 7
> #                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|
> ports: in  0, out  0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6| 7|
> 
> smpquery vlarb  7
> # VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8
> # Low priority VL Arbitration Table:
> VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
> # High priority VL Arbitration Table:
> VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
> WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 |
> 
> Low priority table entries are all zero to skip.
> High priority table give VL 0 and VL 2 different weight.
> 
> The SL is specified on command line, one thread with SL 0, the other
> thread with SL 2.
> 
> Thanks for looking into this, and let me know if more info is needed.

What's the limit of high priority ?

-- Hal

> Feiyi
> 
> 
> 
> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com] 
> Sent: Friday, November 03, 2006 3:27 PM
> To: Wang, Feiyi
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] question on QoS support
> 
> On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote:
> > In our test at the ORNL - it appears you can "turn off" the traffic by
> > giving every VL weight 0.
> 
> A weight of 0 indicates to skip that entry.
> 
> >  As soon as you assign non-zero VL weight,
> > the traffic starts to flow, however, VL with more weight doesn't have
> > expected preference treatment. In other words, traffic shaping didn't
> > take place. smpquery vlarb verified the mapping table was there.
> 
> correctly ?
> 
> Is it high or low priority or both ?
> 
> What about SL2VLMapping table ? Is it setup correctly ?
> 
> What's your topology for this ?
> 
> Can you send your SL2VLMapping and VLarbitration configuration ?
> 
> > I believe the scenario described below 'should' be able to generate
> > congestion point ... but it would be helpful if someone can elaborate
> > a way to "look into" how/if scheduling/arbitration take place.
> 
> The only ways I know would be to look at either the packets on the wire
> or what you are doing with multiple streams which seems valid to me.
> 
> Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to
> understand how to configure this ?
> 
> -- Hal
> 
> > Best,
> > 
> > Feiyi
> > 
> > 
> > On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <halr at voltaire.com>
> wrote:
> > > Hi Oliver,
> > >
> > > On Thu, 2006-11-02 at 10:20, Oliver wrote:
> > > > Hi, Hal -
> > > >
> > > > > How is this being observed/measured ?
> > > >
> > > > Host A, B, with 4x DDR both connected to Flextronic switch.
> > > > A single process of ibv_read_bw gives about 1415MB /s average
> > > > bandwidth. Two concurrent process report 714.45 MB/s each, dead
> even.
> > > > Now if I bump up one process with a different SL, then I expect to
> see
> > > > shaping to take place. Please let me if the scenario makes sense.
> > >
> > > It makes sense. However, if the higher priority traffic does not
> fill
> > > the scheduling, the low priority can take up the slack so I'm not
> sure
> > > if this is what you are seeing or something else.
> > >
> > > It might be interesting to try the same thing at SDR speeds.
> > >
> > > -- Hal
> > >
> > > > > Yes, 8 VLs should be supported in your subnet. You can verify
> this with
> > > > > smpquery portinfo on the HCA port and examine OperVLs assuming
> the port
> > > > > is ACTIVE.
> > > >
> > > > yes, I verified the data VL support, it is 8. I will poke for more
> > > > info with suggested commands by Sasha.
> > > >
> > > > > > A related question is, if I modify qos setting in SM, do I
> need to
> > > > > > restart SA on each hosts for it to see the changes? (I am
> hoping not,
> > > > > > as I tried in the test, it doesn't seem to make a difference)
> > > > >
> > > > > Not sure what you mean. SA is tightly coupled with the OpenSM.
> Do you
> > > > > mean SA client ? The client hosts don't need restarting but did
> you
> > > > > restart OpenSM with your QoS configuration ?
> > > >
> > > > I mean client SA. yes, I understand OpenSM needs to be restarted.
> > > >
> > > > > BTW, which OpenSM are you running ?
> > > >
> > > > OFED 1.1 based.
> > > >
> > > > thanks
> > > >
> > > > - Oliver
> > >
> > >
>