<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7036.0">
<TITLE>Multi-port testing with ibv_rdma_bw shows strange results</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>
<P><FONT SIZE=2 FACE="Arial">I have 2 servers (cst-53, cst-54) connected via one switch using a mlx4 dual port adapter. I did a quick test, using ibv_rdma_bw, across port 1 and then port 2. When running across port 1 the data seems to be split across port 1 and port 2, across port 2 the traffic is all on port 2 as expected. Any ideas? Can I trust perfquery results?</FONT></P>
<P><FONT SIZE=2 FACE="Arial">Thanks,</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">-arlin</FONT>
</P>
<P><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">my configuration (OFED 1.3, RHEL5.1)</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">:</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">cst-53:</FONT></B><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-54 sbin]# ibstat<BR>
CA 'mlx4_0'<BR>
CA type: MT26418<BR>
Number of ports: 2<BR>
Firmware version: 2.4.938<BR>
Hardware version: a0<BR>
Node GUID: 0x0002c9030000a5b4<BR>
System image GUID: 0x0002c9030000a5b7<BR>
Port 1:<BR>
State: Active<BR>
Physical state: LinkUp<BR>
Rate: 10<BR>
Base lid: 24<BR>
LMC: 0<BR>
SM lid: 3<BR>
Capability mask: 0x02510868<BR>
Port GUID: 0x0002c9030000a5b5<BR>
Port 2:<BR>
State: Active<BR>
Physical state: LinkUp<BR>
Rate: 10<BR>
Base lid: 25<BR>
LMC: 0<BR>
SM lid: 3<BR>
Capability mask: 0x02510868<BR>
Port GUID: 0x0002c9030000a5b6</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">cst-54:</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-53 fw-25408-rel-2_4_938]# ibstat<BR>
CA 'mlx4_0'<BR>
CA type: MT26418<BR>
Number of ports: 2<BR>
Firmware version: 2.4.938<BR>
Hardware version: a0<BR>
Node GUID: 0x0002c9030000a620<BR>
System image GUID: 0x0002c9030000a623<BR>
Port 1:<BR>
State: Active<BR>
Physical state: LinkUp<BR>
Rate: 2<BR>
Base lid: 22<BR>
LMC: 0<BR>
SM lid: 3<BR>
Capability mask: 0x02510868<BR>
Port GUID: 0x0002c9030000a621<BR>
Port 2:<BR>
State: Active<BR>
Physical state: LinkUp<BR>
Rate: 10<BR>
Base lid: 23<BR>
LMC: 0<BR>
SM lid: 3<BR>
Capability mask: 0x02510868<BR>
Port GUID: 0x0002c9030000a622</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">TEST results:</FONT></B><FONT FACE="Times New Roman"> </FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ibv_rdma_bw on port 2 is working fine, all traffic on port 2:</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">:</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">server:</FONT></B><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">perfquery -R;/usr/bin/ib_rdma_bw -i 2;perfquery 24 1;perfquery 25 2</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">11621: | port=18515 |</FONT><B> <FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ib_port=2</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |<BR>
11621: Local address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6 RKey 0xc8002401 VAddr 0x002aaaaaf03000<BR>
11621: Remote address: LID 0x17, QPN 0x16004a, PSN 0x65136b, RKey 0x82002401 VAddr 0x002aaaab313000</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"># Port counters: Lid 24 port 1<BR>
PortSelect:......................1<BR>
XmtData:.........................0<BR>
RcvData:.........................0<BR>
XmtPkts:.........................0<BR>
RcvPkts:.........................0<BR>
# Port counters: Lid 25 port 2<BR>
PortSelect:......................2<BR>
XmtData:.........................62946<BR>
RcvData:.........................116072668<BR>
XmtPkts:.........................7279<BR>
RcvPkts:.........................224274</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">client:</FONT></B><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">perfquery -R;/usr/bin/ib_rdma_bw -i 2 cst-54;perfquery 22 1;perfquery 23 2</FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">9190: | port=18515 |</FONT><B> <FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ib_port=2</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |<BR>
9190: Local address: LID 0x17, QPN 0x16004a, PSN 0x65136b RKey 0x82002401 VAddr 0x002aaaab313000<BR>
9190: Remote address: LID 0x19, QPN 0x16004a, PSN 0x42d3c6, RKey 0xc8002401 VAddr 0x002aaaaaf03000</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">9190: Bandwidth peak (#0 to #983): 937.621 MB/sec<BR>
9190: Bandwidth average: 937.614 MB/sec<BR>
9190: Service Demand peak (#0 to #983): 2077 cycles/KB<BR>
9190: Service Demand Avg : 2077 cycles/KB<BR>
# Port counters: Lid 22 port 1<BR>
XmtData:.........................0<BR>
RcvData:.........................0<BR>
XmtPkts:.........................0<BR>
RcvPkts:.........................0<BR>
# Port counters: Lid 23 port 2<BR>
XmtData:.........................116075478<BR>
RcvData:.........................66442<BR>
XmtPkts:.........................224298<BR>
RcvPkts:.........................7318</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">port 1 with strange results - traffic split between port 1 and 2</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">: </FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">client:</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-53 fw-25408-rel-2_4_938]# perfquery -R;/usr/bin/ib_rdma_bw -i 1 cst-54;perfquery 22 1;perfquery 23 2 </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">9144: | port=18515 |</FONT><B> <FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ib_port=1</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |<BR>
9144: Local address: LID 0x16, QPN 0x10004a, PSN 0xd125be RKey 0x34002401 VAddr 0x002aaaab313000<BR>
9144: Remote address: LID 0x18, QPN 0x10004a, PSN 0x5cb52c, RKey 0x7a002401 VAddr 0x002aaaaaf03000 </FONT>
<BR>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">9144: Bandwidth peak (#0 to #978): 234.634 MB/sec<BR>
9144: Bandwidth average: 234.634 MB/sec<BR>
9144: Service Demand peak (#0 to #978): 8303 cycles/KB<BR>
9144: Service Demand Avg : 8303 cycles/KB</FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"># Port counters: Lid 22 port 1<BR>
XmtData:.........................16580072<BR>
RcvData:.........................7072<BR>
XmtPkts:.........................32001<BR>
RcvPkts:.........................1001<BR>
# Port counters: Lid 23 port 2<BR>
XmtData:.........................82915046<BR>
RcvData:.........................51692<BR>
XmtPkts:.........................160292<BR>
RcvPkts:.........................5308</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">server:</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-54 sbin]# perfquery -R;/usr/bin/ib_rdma_bw -i 1 ;perfquery 24 1;perfquery 25 2<BR>
11586: | port=18515 |</FONT><B> <FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ib_port=1</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |<BR>
11586: Local address: LID 0x18, QPN 0x14004a, PSN 0x5e8160 RKey 0xae002401 VAddr 0x002aaaaaf03000<BR>
11586: Remote address: LID 0x16, QPN 0x14004a, PSN 0xf0dd70, RKey 0x68002401 VAddr 0x002aaaab313000 </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"># Port counters: Lid 24 port 1<BR>
XmtData:.........................7000<BR>
RcvData:.........................16580000<BR>
XmtPkts:.........................1000<BR>
RcvPkts:.........................32000<BR>
# Port counters: Lid 25 port 2<BR>
XmtData:.........................55802<BR>
RcvData:.........................99492206<BR>
XmtPkts:.........................6277<BR>
RcvPkts:.........................192268</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">ibtracert:</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">cst-53 port 1 to cst-54 port 1</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-54 sbin]# ibtracert 22 24<BR>
>From ca {0x0002c9030000a620} portnum 1 lid 22-22 "cst-53 HCA-1"<BR>
[1] -> switch port {0x000b8cffff004046}[2] lid 2-2 "MT47396 Infiniscale-III Mellanox Technologies"<BR>
[4] -> ca port {0x0002c9030000a5b5}[1] lid 24-24 "cst-54 HCA-1"<BR>
To ca {0x0002c9030000a5b4} portnum 1 lid 24-24 "cst-54 HCA-1"</FONT>
<BR><B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">cst-53 port 2 to cst-54 port 2</FONT></B><FONT COLOR="#0000FF" SIZE=2 FACE="Arial"> </FONT>
<BR><FONT COLOR="#0000FF" SIZE=2 FACE="Arial">[root@cst-54 sbin]# ibtracert 23 25<BR>
>From ca {0x0002c9030000a620} portnum 2 lid 23-23 "cst-53 HCA-1"<BR>
[2] -> switch port {0x000b8cffff004046}[3] lid 2-2 "MT47396 Infiniscale-III Mellanox Technologies"<BR>
[1] -> ca port {0x0002c9030000a5b6}[2] lid 25-25 "cst-54 HCA-1"<BR>
To ca {0x0002c9030000a5b4} portnum 2 lid 25-25 "cst-54 HCA-1"<BR>
</FONT>
<BR><FONT SIZE=2 FACE="Arial"> </FONT>
</P>
</BODY>
</HTML>