[openib-general] question on QoS support

[Wang, Feiyi]

The test is done on two hosts, say A and B. A has 4x SDR (run ib_rdam_bw
as server), B has 4x DDR (run more than one thread of ib_rdma_bw as
clients). The sl2vl table read as:

smpquery sl2vl 7
# SL2VL table: Lid 7
#                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|
ports: in  0, out  0: | 0| 1| 2| 3| 4| 5| 6| 7| 0| 1| 2| 3| 4| 5| 6| 7|

smpquery vlarb  7
# VLArbitration tables: Lid 7 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL    : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x6 |0x7 |
WEIGHT: |0x1 |0x0 |0x8 |0x0 |0x0 |0x0 |0x0 |0x0 |

Low priority table entries are all zero to skip.
High priority table give VL 0 and VL 2 different weight.

The SL is specified on command line, one thread with SL 0, the other
thread with SL 2.

Thanks for looking into this, and let me know if more info is needed.

Feiyi



-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com] 
Sent: Friday, November 03, 2006 3:27 PM
To: Wang, Feiyi
Cc: openib-general at openib.org
Subject: Re: [openib-general] question on QoS support

On Fri, 2006-11-03 at 15:12, Feiyi Wang wrote:
> In our test at the ORNL - it appears you can "turn off" the traffic by
> giving every VL weight 0.

A weight of 0 indicates to skip that entry.

>  As soon as you assign non-zero VL weight,
> the traffic starts to flow, however, VL with more weight doesn't have
> expected preference treatment. In other words, traffic shaping didn't
> take place. smpquery vlarb verified the mapping table was there.

correctly ?

Is it high or low priority or both ?

What about SL2VLMapping table ? Is it setup correctly ?

What's your topology for this ?

Can you send your SL2VLMapping and VLarbitration configuration ?

> I believe the scenario described below 'should' be able to generate
> congestion point ... but it would be helpful if someone can elaborate
> a way to "look into" how/if scheduling/arbitration take place.

The only ways I know would be to look at either the packets on the wire
or what you are doing with multiple streams which seems valid to me.

Have you read section 7.6.9.2 (p. 189-190) in IBA 1.2 volume 1 to
understand how to configure this ?

-- Hal

> Best,
> 
> Feiyi
> 
> 
> On 02 Nov 2006 10:49:04 -0500, Hal Rosenstock <halr at voltaire.com>
wrote:
> > Hi Oliver,
> >
> > On Thu, 2006-11-02 at 10:20, Oliver wrote:
> > > Hi, Hal -
> > >
> > > > How is this being observed/measured ?
> > >
> > > Host A, B, with 4x DDR both connected to Flextronic switch.
> > > A single process of ibv_read_bw gives about 1415MB /s average
> > > bandwidth. Two concurrent process report 714.45 MB/s each, dead
even.
> > > Now if I bump up one process with a different SL, then I expect to
see
> > > shaping to take place. Please let me if the scenario makes sense.
> >
> > It makes sense. However, if the higher priority traffic does not
fill
> > the scheduling, the low priority can take up the slack so I'm not
sure
> > if this is what you are seeing or something else.
> >
> > It might be interesting to try the same thing at SDR speeds.
> >
> > -- Hal
> >
> > > > Yes, 8 VLs should be supported in your subnet. You can verify
this with
> > > > smpquery portinfo on the HCA port and examine OperVLs assuming
the port
> > > > is ACTIVE.
> > >
> > > yes, I verified the data VL support, it is 8. I will poke for more
> > > info with suggested commands by Sasha.
> > >
> > > > > A related question is, if I modify qos setting in SM, do I
need to
> > > > > restart SA on each hosts for it to see the changes? (I am
hoping not,
> > > > > as I tried in the test, it doesn't seem to make a difference)
> > > >
> > > > Not sure what you mean. SA is tightly coupled with the OpenSM.
Do you
> > > > mean SA client ? The client hosts don't need restarting but did
you
> > > > restart OpenSM with your QoS configuration ?
> > >
> > > I mean client SA. yes, I understand OpenSM needs to be restarted.
> > >
> > > > BTW, which OpenSM are you running ?
> > >
> > > OFED 1.1 based.
> > >
> > > thanks
> > >
> > > - Oliver
> >
> >