[ofa-general] Multi-port testing with ibv_rdma_bw shows strange results

Thu May 22 14:55:17 PDT 2008

I have 2 servers (cst-53, cst-54) connected via one switch using a mlx4 dual port adapter. I did a
quick test, using ibv_rdma_bw, across port 1 and then port 2. When running across port 1 the data
seems to be split across port 1 and port 2, across port 2 the traffic is all on port 2 as expected.
Any ideas? Can I trust perfquery results?

Thanks,

-arlin

my configuration (OFED 1.3, RHEL5.1): 
cst-53: 
[root at cst-54 sbin]# ibstat
CA 'mlx4_0'
CA type: MT26418
Number of ports: 2
Firmware version: 2.4.938
Hardware version: a0
Node GUID: 0x0002c9030000a5b4
System image GUID: 0x0002c9030000a5b7
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 24
LMC: 0
SM lid: 3
Capability mask: 0x02510868
Port GUID: 0x0002c9030000a5b5
Port 2:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 25
LMC: 0
SM lid: 3
Capability mask: 0x02510868
Port GUID: 0x0002c9030000a5b6
cst-54: 
[root at cst-53 fw-25408-rel-2_4_938]# ibstat
CA 'mlx4_0'
CA type: MT26418
Number of ports: 2
Firmware version: 2.4.938
Hardware version: a0
Node GUID: 0x0002c9030000a620
System image GUID: 0x0002c9030000a623
Port 1:
State: Active
Physical state: LinkUp
Rate: 2
Base lid: 22
LMC: 0
SM lid: 3
Capability mask: 0x02510868
Port GUID: 0x0002c9030000a621
Port 2:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 23
LMC: 0
SM lid: 3
Capability mask: 0x02510868
Port GUID: 0x0002c9030000a622
TEST results: 
ibv_rdma_bw on port 2 is working fine, all traffic on port 2:: 
server: 
perfquery -R;/usr/bin/ib_rdma_bw -i 2;perfquery 24 1;perfquery 25 2 
11621: | port=18515 | ib_port=2 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
11621: Local address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6 RKey 0xc8002401 VAddr 0x002aaaaaf03000
11621: Remote address: LID 0x17, QPN 0x16004a, PSN 0x65136b, RKey 0x82002401 VAddr 0x002aaaab313000 
# Port counters: Lid 24 port 1
PortSelect:......................1
XmtData:.........................0
RcvData:.........................0
XmtPkts:.........................0
RcvPkts:.........................0
# Port counters: Lid 25 port 2
PortSelect:......................2
XmtData:.........................62946
RcvData:.........................116072668
XmtPkts:.........................7279
RcvPkts:.........................224274
client: 
perfquery -R;/usr/bin/ib_rdma_bw -i 2 cst-54;perfquery 22 1;perfquery 23 2
9190: | port=18515 | ib_port=2 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
9190: Local address: LID 0x17, QPN 0x16004a, PSN 0x65136b RKey 0x82002401 VAddr 0x002aaaab313000
9190: Remote address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6, RKey 0xc8002401 VAddr 0x002aaaaaf03000 
9190: Bandwidth peak (#0 to #983): 937.621 MB/sec
9190: Bandwidth average: 937.614 MB/sec
9190: Service Demand peak (#0 to #983): 2077 cycles/KB
9190: Service Demand Avg : 2077 cycles/KB
# Port counters: Lid 22 port 1
XmtData:.........................0
RcvData:.........................0
XmtPkts:.........................0
RcvPkts:.........................0
# Port counters: Lid 23 port 2
XmtData:.........................116075478
RcvData:.........................66442
XmtPkts:.........................224298
RcvPkts:.........................7318
port 1 with strange results - traffic split between port 1 and 2: 
client: 
[root at cst-53 fw-25408-rel-2_4_938]# perfquery -R;/usr/bin/ib_rdma_bw -i 1 cst-54;perfquery 22
1;perfquery 23 2 
9144: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
9144: Local address: LID 0x16, QPN 0x10004a, PSN 0xd125be RKey 0x34002401 VAddr 0x002aaaab313000
9144: Remote address: LID 0x18, QPN 0x10004a, PSN 0x5cb52c, RKey 0x7a002401 VAddr 0x002aaaaaf03000 

9144: Bandwidth peak (#0 to #978): 234.634 MB/sec
9144: Bandwidth average: 234.634 MB/sec
9144: Service Demand peak (#0 to #978): 8303 cycles/KB
9144: Service Demand Avg : 8303 cycles/KB
# Port counters: Lid 22 port 1
XmtData:.........................16580072
RcvData:.........................7072
XmtPkts:.........................32001
RcvPkts:.........................1001
# Port counters: Lid 23 port 2
XmtData:.........................82915046
RcvData:.........................51692
XmtPkts:.........................160292
RcvPkts:.........................5308
server: 
[root at cst-54 sbin]# perfquery -R;/usr/bin/ib_rdma_bw -i 1 ;perfquery 24 1;perfquery 25 2
11586: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
11586: Local address: LID 0x18, QPN 0x14004a, PSN 0x5e8160 RKey 0xae002401 VAddr 0x002aaaaaf03000
11586: Remote address: LID 0x16, QPN 0x14004a, PSN 0xf0dd70, RKey 0x68002401 VAddr 0x002aaaab313000 
# Port counters: Lid 24 port 1
XmtData:.........................7000
RcvData:.........................16580000
XmtPkts:.........................1000
RcvPkts:.........................32000
# Port counters: Lid 25 port 2
XmtData:.........................55802
RcvData:.........................99492206
XmtPkts:.........................6277
RcvPkts:.........................192268
ibtracert: 
cst-53 port 1 to cst-54 port 1 
[root at cst-54 sbin]# ibtracert 22 24
>From ca {0x0002c9030000a620} portnum 1 lid 22-22 "cst-53 HCA-1"
[1] -> switch port {0x000b8cffff004046}[2] lid 2-2 "MT47396 Infiniscale-III Mellanox Technologies"
[4] -> ca port {0x0002c9030000a5b5}[1] lid 24-24 "cst-54 HCA-1"
To ca {0x0002c9030000a5b4} portnum 1 lid 24-24 "cst-54 HCA-1"
cst-53 port 2 to cst-54 port 2 
[root at cst-54 sbin]# ibtracert 23 25
>From ca {0x0002c9030000a620} portnum 2 lid 23-23 "cst-53 HCA-1"
[2] -> switch port {0x000b8cffff004046}[3] lid 2-2 "MT47396 Infiniscale-III Mellanox Technologies"
[1] -> ca port {0x0002c9030000a5b6}[2] lid 25-25 "cst-54 HCA-1"
To ca {0x0002c9030000a5b4} portnum 2 lid 25-25 "cst-54 HCA-1"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080522/c01749bf/attachment.html>