[ofa-general] QoS implementation

[Celine Bourde]

Hi,

I'm testing QoS on opensm.
I work with OFED-1.4-20081123-0600.tgz and opensm-3.2.4_20081122_c732c34.
I've set up qos-policy file, SL2VL and VLArbitration Table (all in 
attachement).
Results still have wrong values.

I've launch opensm -Q /etc/ofa/opensm.conf and use qperf tools to test 
QoS implementation :

bandwith test :
---------------
SL1 should have 33% of bandwidth (1:64) , SL2 sould have 66% of 
bandwidth (2:128)

cmd server :
qperf -lp 19766 & qperf -lp 19764

cmd client :
qperf 192.168.0.3 -lp 19766 -sl 1 rc_rdma_write_bw > sl1.txt & qperf 
192.168.0.3 -lp 19764 -sl 2 rc_rdma_write_bw > sl2.txt

results :
sl1.txt : rc_rdma_write_bw:
    bw  =  1.7 GB/sec

sl2.txt : rc_rdma_write_bw:
    bw  =  1.7 GB/sec

latency test :
--------------
cmd server:
qperf -lp 19766 & qperf -lp 19764

cmd client :
qperf 192.168.0.3 -lp 19766 -sl 1 rc_rdma_write_lat > sl1.txt & qperf 
192.168.0.3 -lp 19764 -sl 2 rc_rdma_write_lat > sl2.txt

results :
sl1.txt : rc_rdma_write_lat
        latency  =  12.9 us
sl2.txt : rc_rdma_write_lat:
        latency  =  12.9 us

I've tested pc to pc without switch between both.
I've a Mellanox ConnectX card on each pc with following features :
ConnectX® IB QDR, ConnectX IB HCA/TCA IC, dual-port, QDR, PCIe 2.0
PCIe 2.0 5.0GT/s

Firmware has been updated with latest version 2.5.9

[]# ibstat
CA 'mlx4_0'
   CA type: MT26428
   Number of ports: 2
   Firmware version: 2.5.900
   Hardware version: a0
   Node GUID: 0x0002c903000290aa
   System image GUID: 0x0002c903000290ad
   Capability mask: 0x0251086a


When I use smpquery, results are the following :

VLCap:...........................VL0-7
VLHighLimit:.....................0
VLArbHighCap:....................8
VLArbLowCap:.....................8
VLStallCount:....................0

So, my configuration (cf opensm.conf in attachement) doesn't correspond 
to smpquery results.
I've tried to restart openibd on both pc. I've restarted opensm (stop 
and start),
I'v tested "opensm -Q conf_file" but my configuration is always unset.

Did I miss something or is it a bug ?

Thanks for your help.

Céline Bourde.


MY CONFIGURATION IS THE FOLLOWING :

I've added "options mlx4_core enable_qos=1" in modprobe.conf to set on QoS

I've configured qos-policy file with following rules :
-----------------------
qos-levels

      qos-level
              name: DEFAULT
              sl: 0
      end-qos-level

      qos-level
              name: MPI
              sl: 1
      end-qos-level

      qos-level
              name: Lustre
              sl: 2
      end-qos-level


end-qos-levels
----------------------


my qos settings in /etc/ofa/opensm.conf
-----------------------
#
# QoS OPTIONS
#
# Enable QoS setup
qos TRUE

# QoS policy file to be used
qos_policy_file /etc/opensm/qos-policy.conf

# QoS default options
qos_max_vls 15
qos_high_limit 0
qos_vlarb_high 
0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_vlarb_low 
0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

# QoS CA options
qos_ca_max_vls 15
qos_ca_high_limit 0
qos_ca_vlarb_high 0:0,1:0,2:0
qos_ca_vlarb_low 0:1,1:64,2:128
qos_ca_sl2vl 0,1,2,3,4,6,7,8,9,10,11,12,13,14,7,5

# QoS Switch external ports options
qos_swe_max_vls 15
qos_swe_high_limit 0
qos_swe_vlarb_high 0:0,1:0,2:0
qos_swe_vlarb_low 0:1,1:64,2:128
qos_swe_sl2vl 0,1,2,3,4,6,7,8,9,10,11,12,13,14,7,5