[ofa-general] Multi-port testing with ibv_rdma_bw shows strangeresults

Davis, Arlin R arlin.r.davis at intel.com
Thu May 22 15:42:36 PDT 2008


never mind. my use of perfquery to reset counters was not correct. 


________________________________

	From: general-bounces at lists.openfabrics.org
[mailto:general-bounces at lists.openfabrics.org] On Behalf Of Davis, Arlin
R
	Sent: Thursday, May 22, 2008 2:55 PM
	To: [ofa_general]
	Subject: [ofa-general] Multi-port testing with ibv_rdma_bw shows
strangeresults
	
	


	I have 2 servers (cst-53, cst-54) connected via one switch using
a mlx4 dual port adapter. I did a quick test, using ibv_rdma_bw, across
port 1 and then port 2. When running across port 1 the data seems to be
split across port 1 and port 2, across port 2 the traffic is all on port
2 as expected. Any ideas? Can I trust perfquery results?

	Thanks, 

	-arlin 

	my configuration (OFED 1.3, RHEL5.1): 
	cst-53: 
	[root at cst-54 sbin]# ibstat
	CA 'mlx4_0'
	CA type: MT26418
	Number of ports: 2
	Firmware version: 2.4.938
	Hardware version: a0
	Node GUID: 0x0002c9030000a5b4
	System image GUID: 0x0002c9030000a5b7
	Port 1:
	State: Active
	Physical state: LinkUp
	Rate: 10
	Base lid: 24
	LMC: 0
	SM lid: 3
	Capability mask: 0x02510868
	Port GUID: 0x0002c9030000a5b5
	Port 2:
	State: Active
	Physical state: LinkUp
	Rate: 10
	Base lid: 25
	LMC: 0
	SM lid: 3
	Capability mask: 0x02510868
	Port GUID: 0x0002c9030000a5b6 
	cst-54: 
	[root at cst-53 fw-25408-rel-2_4_938]# ibstat
	CA 'mlx4_0'
	CA type: MT26418
	Number of ports: 2
	Firmware version: 2.4.938
	Hardware version: a0
	Node GUID: 0x0002c9030000a620
	System image GUID: 0x0002c9030000a623
	Port 1:
	State: Active
	Physical state: LinkUp
	Rate: 2
	Base lid: 22
	LMC: 0
	SM lid: 3
	Capability mask: 0x02510868
	Port GUID: 0x0002c9030000a621
	Port 2:
	State: Active
	Physical state: LinkUp
	Rate: 10
	Base lid: 23
	LMC: 0
	SM lid: 3
	Capability mask: 0x02510868
	Port GUID: 0x0002c9030000a622 
	TEST results: 
	ibv_rdma_bw on port 2 is working fine, all traffic on port 2:: 
	server: 
	perfquery -R;/usr/bin/ib_rdma_bw -i 2;perfquery 24 1;perfquery
25 2 
	11621: | port=18515 | ib_port=2 | size=65536 | tx_depth=100 |
iters=1000 | duplex=0 | cma=0 |
	11621: Local address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6 RKey
0xc8002401 VAddr 0x002aaaaaf03000
	11621: Remote address: LID 0x17, QPN 0x16004a, PSN 0x65136b,
RKey 0x82002401 VAddr 0x002aaaab313000 
	# Port counters: Lid 24 port 1
	PortSelect:......................1
	XmtData:.........................0
	RcvData:.........................0
	XmtPkts:.........................0
	RcvPkts:.........................0
	# Port counters: Lid 25 port 2
	PortSelect:......................2
	XmtData:.........................62946
	RcvData:.........................116072668
	XmtPkts:.........................7279
	RcvPkts:.........................224274 
	client: 
	perfquery -R;/usr/bin/ib_rdma_bw -i 2 cst-54;perfquery 22
1;perfquery 23 2 
	9190: | port=18515 | ib_port=2 | size=65536 | tx_depth=100 |
iters=1000 | duplex=0 | cma=0 |
	9190: Local address: LID 0x17, QPN 0x16004a, PSN 0x65136b RKey
0x82002401 VAddr 0x002aaaab313000
	9190: Remote address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6, RKey
0xc8002401 VAddr 0x002aaaaaf03000 
	9190: Bandwidth peak (#0 to #983): 937.621 MB/sec
	9190: Bandwidth average: 937.614 MB/sec
	9190: Service Demand peak (#0 to #983): 2077 cycles/KB
	9190: Service Demand Avg : 2077 cycles/KB
	# Port counters: Lid 22 port 1
	XmtData:.........................0
	RcvData:.........................0
	XmtPkts:.........................0
	RcvPkts:.........................0
	# Port counters: Lid 23 port 2
	XmtData:.........................116075478
	RcvData:.........................66442
	XmtPkts:.........................224298
	RcvPkts:.........................7318 
	port 1 with strange results - traffic split between port 1 and
2: 
	client: 
	[root at cst-53 fw-25408-rel-2_4_938]# perfquery
-R;/usr/bin/ib_rdma_bw -i 1 cst-54;perfquery 22 1;perfquery 23 2 
	9144: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 |
iters=1000 | duplex=0 | cma=0 |
	9144: Local address: LID 0x16, QPN 0x10004a, PSN 0xd125be RKey
0x34002401 VAddr 0x002aaaab313000
	9144: Remote address: LID 0x18, QPN 0x10004a, PSN 0x5cb52c, RKey
0x7a002401 VAddr 0x002aaaaaf03000 
	
	9144: Bandwidth peak (#0 to #978): 234.634 MB/sec
	9144: Bandwidth average: 234.634 MB/sec
	9144: Service Demand peak (#0 to #978): 8303 cycles/KB
	9144: Service Demand Avg : 8303 cycles/KB 
	# Port counters: Lid 22 port 1
	XmtData:.........................16580072
	RcvData:.........................7072
	XmtPkts:.........................32001
	RcvPkts:.........................1001
	# Port counters: Lid 23 port 2
	XmtData:.........................82915046
	RcvData:.........................51692
	XmtPkts:.........................160292
	RcvPkts:.........................5308 
	server: 
	[root at cst-54 sbin]# perfquery -R;/usr/bin/ib_rdma_bw -i 1
;perfquery 24 1;perfquery 25 2
	11586: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 |
iters=1000 | duplex=0 | cma=0 |
	11586: Local address: LID 0x18, QPN 0x14004a, PSN 0x5e8160 RKey
0xae002401 VAddr 0x002aaaaaf03000
	11586: Remote address: LID 0x16, QPN 0x14004a, PSN 0xf0dd70,
RKey 0x68002401 VAddr 0x002aaaab313000 
	# Port counters: Lid 24 port 1
	XmtData:.........................7000
	RcvData:.........................16580000
	XmtPkts:.........................1000
	RcvPkts:.........................32000
	# Port counters: Lid 25 port 2
	XmtData:.........................55802
	RcvData:.........................99492206
	XmtPkts:.........................6277
	RcvPkts:.........................192268 
	ibtracert: 
	cst-53 port 1 to cst-54 port 1 
	[root at cst-54 sbin]# ibtracert 22 24
	>From ca {0x0002c9030000a620} portnum 1 lid 22-22 "cst-53 HCA-1"
	[1] -> switch port {0x000b8cffff004046}[2] lid 2-2 "MT47396
Infiniscale-III Mellanox Technologies"
	[4] -> ca port {0x0002c9030000a5b5}[1] lid 24-24 "cst-54 HCA-1"
	To ca {0x0002c9030000a5b4} portnum 1 lid 24-24 "cst-54 HCA-1" 
	cst-53 port 2 to cst-54 port 2 
	[root at cst-54 sbin]# ibtracert 23 25
	>From ca {0x0002c9030000a620} portnum 2 lid 23-23 "cst-53 HCA-1"
	[2] -> switch port {0x000b8cffff004046}[3] lid 2-2 "MT47396
Infiniscale-III Mellanox Technologies"
	[1] -> ca port {0x0002c9030000a5b6}[2] lid 25-25 "cst-54 HCA-1"
	To ca {0x0002c9030000a5b4} portnum 2 lid 25-25 "cst-54 HCA-1"
	
	  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080522/c7e6d0b5/attachment.html>


More information about the general mailing list