<div dir="ltr"><div>PortXmitWait is a sign of congestion on those ports. It's the rate of increase (and on which "tier" in subnet it is occurring) that matters. Note that IB counters are sticky rather than rollover. It is 32 bit counter. infiniband-diags gives raw numbers. There are some proprietary tools (e.g. UFM) that do better analysis on this.</div><div><br></div><div>As to multiple OpenSMs, this should be fine. One will be elected master; others should be in standby. Hopefully, all the SMs are identical including configuration.</div><div><br></div><div>-- Hal</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 25, 2015 at 7:29 AM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Hi all,<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">I'm having some issues with my IB network, basically I have the following setup (pdf attach). I run a fio test between the HP Blade with QDR (bonding ports active/backup mode), to a storage cluster with FDR (with no bonding at all), and the best result that I can get is 1.7GB/s, that's pretty slow actually. However I was hopping something between 2.5-3.5GB/s on a QDR infiniband network. Then I try to tweak some parameters, for example setting the scaling_governor to 'performance', and set the 'connected' mode in the ib ports, then change the following variables:<br><br>sysctl -w net.core.netdev_max_backlog=250000<br>sysctl -w net.core.rmem_max=4194304<br>sysctl -w net.core.wmem_max=4194304<br>sysctl -w net.core.rmem_default=4194304<br>sysctl -w net.core.wmem_default=4194304<br>sysctl -w net.core.optmem_max=4194304<br>sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"<br>sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304"<br>sysctl -w net.ipv4.tcp_low_latency=1<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">The bond configuration is the following:<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># cat /etc/modprobe.d/bonding.conf<br><br>alias bond-ib bonding options bonding mode=1 miimon=100 downdelay=100 updelay=100 max_bonds=2<br><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># cat /etc/network/interfaces<br><br>(...)<br><br>## INFINIBAND CONF<br>auto ib0<br>iface ib0 inet manual<br> bond-master bond-ib<br><br>auto ib1<br>iface ib1 inet manual<br> bond-master bond-ib<br><br>auto bond-ib<br>iface bond-ib inet static<br> address 172.23.18.1<br> netmask 255.255.240.0<br> slaves ib0 ib1<br> bond_miimon 100<br> bond_mode active-backup<br> pre-up echo connected > /sys/class/net/ib0/mode<br> pre-up echo connected > /sys/class/net/ib1/mode<br> pre-up /sbin/ifconfig ib0 mtu 65520<br> pre-up /sbin/ifconfig ib1 mtu 65520<br> pre-up modprobe bond-ib<br> pre-up /sbin/ifconfig bond-ib mtu 65520<br><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">OS is Ubuntu 14.04.3 LTS on the HP blade with Kernel 3.13.0-63-generic, and Ubuntu 14.04.3 LTS with kernel 3.19.0-25-generic for the storage cluster.<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">The IB Mezzanine cards on the HP Blades are "InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)". And on the storage cluster the IB ADPT are "Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]<br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Then I run in one of the nodes in cluster the '<b>ibqueryerrors</b>' command and found the following:<br><br>$ ibqueryerrors<br>Errors for "e60-host01 HCA-1" <span style="color:rgb(255,0,0)">---> blade1 the one with the bonding configuration using internally HP-IB-SW port 17 and 25</span><br> GUID 0xf452140300dd3296 port 2: [PortXmitWait == 15]<br>Errors for 0x2c902004b0918 "Infiniscale-IV Mellanox Technologies"<br> GUID 0x2c902004b0918 port ALL: [PortXmitWait == 325727936]<br> GUID 0x2c902004b0918 port 25: [PortXmitWait == 325727936]<br>Errors for 0xe41d2d030031e9c1 "MF0;GWIB01:SX6036G/U1"<br> GUID 0xe41d2d030031e9c1 port ALL: [PortXmitWait == 326981305]<br> GUID 0xe41d2d030031e9c1 port 11: [PortXmitWait == 326976642]<br> GUID 0xe41d2d030031e9c1 port 36: [PortXmitWait == 4663]<br>Errors for 0xf45214030073f500 "MF0;SWIB02:SX6018/U1"<br> GUID 0xf45214030073f500 port ALL: [PortXmitWait == 13979524]<br> GUID 0xf45214030073f500 port 8: [PortXmitWait == 3749467]<br> GUID 0xf45214030073f500 port 9: [PortXmitWait == 3434343]<br> GUID 0xf45214030073f500 port 10: [PortXmitWait == 3389114]<br> GUID 0xf45214030073f500 port 11: [PortXmitWait == 3406600]<br>Errors for 0xe41d2d030031eb41 "MF0;GWIB02:SX6036G/U1"<br> GUID 0xe41d2d030031eb41 port ALL: [PortXmitWait == 1352]<br> GUID 0xe41d2d030031eb41 port 34: [PortXmitWait == 1352]<br>Errors for "cibn08 HCA-1"<br> GUID 0xe41d2d03007b77c1 port 1: [PortXmitWait == 813152781]<br> GUID 0xe41d2d03007b77c2 port 2: [PortXmitWait == 3256286]<br>Errors for "cibn07 HCA-1"<br> GUID 0xe41d2d03007b67c1 port 1: [PortXmitWait == 841850209]<br> GUID 0xe41d2d03007b67c2 port 2: [PortXmitWait == 3211488]<br>Errors for "cibn05 HCA-1"<br> GUID 0xe41d2d0300d95191 port 1: [PortXmitWait == 840576923]<br> GUID 0xe41d2d0300d95192 port 2: [PortXmitWait == 2635901]<br>Errors for "cibn06 HCA-1"<br> GUID 0xe41d2d03007b77b1 port 1: [PortXmitWait == 843231930]<br> GUID 0xe41d2d03007b77b2 port 2: [PortXmitWait == 2869022]<br>Errors for 0xe41d2d0300097630 "MF0;SWIB01:SX6018/U1"<br> GUID 0xe41d2d0300097630 port ALL: [PortXmitWait == 470746689]<br> GUID 0xe41d2d0300097630 port 0: [PortXmitWait == 7]<br> GUID 0xe41d2d0300097630 port 2: [PortXmitWait == 8046]<br> GUID 0xe41d2d0300097630 port 3: [PortXmitWait == 7631]<br> GUID 0xe41d2d0300097630 port 8: [PortXmitWait == 219608]<br> GUID 0xe41d2d0300097630 port 9: [PortXmitWait == 216118]<br> GUID 0xe41d2d0300097630 port 10: [PortXmitWait == 198693]<br> GUID 0xe41d2d0300097630 port 11: [PortXmitWait == 206192]<br> GUID 0xe41d2d0300097630 port 18: [PortXmitWait == 469890394]<br>Errors for "cibm01 HCA-1"<br> GUID 0xe41d2d0300163651 port 1: [PortXmitWait == 6002]<br><br>## Summary: 22 nodes checked, 11 bad nodes found<br>## 208 ports checked, 26 ports have errors beyond threshold<br>## <br>## Suppressed:<br><br><br>$ ibportstate -L 29 17 query<br>Switch PortInfo:<br># Port info: Lid 29 port 17<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................75<br>SMLid:...........................2328<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Peer PortInfo:<br># Port info: Lid 29 DR path slid 4; dlid 65535; 0,17 port 1<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................32<br>SMLid:...........................2<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Mkey:............................<not displayed><br>MkeyLeasePeriod:.................0<br>ProtectBits:.....................0<br><br>---<br><br>$ ibportstate -L 29 25 query<br>Switch PortInfo:<br># Port info: Lid 29 port 25<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................75<br>SMLid:...........................2328<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Peer PortInfo:<br># Port info: Lid 29 DR path slid 4; dlid 65535; 0,25 port 2<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................33<br>SMLid:...........................2<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Mkey:............................<not displayed><br>MkeyLeasePeriod:.................0<br>ProtectBits:.....................0<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">First I thought that maybe some cables could be in a bad state, but.. all of them?... so I really don't know if maybe this XmitWait could be pushing some noise on the performance at all or not. Any ideas? or hints? Also I had the SM configured on SWIB01 with high priority and then a second SM configured on SWIB02 with less priority, both in an active state, is this ok? Or is better to only have one and only one SM active at a time in the entire IB network?<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Also find below some iperf tests between blades that are on different enclosures:<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><b>e61-host01 (server):</b><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># iperf -s<br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br><b>e60-host01 (client):</b><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># iperf -c 172.23.18.10 -P 4<br><br>------------------------------------------------------------<br>Client connecting to 172.23.18.10, TCP port 5001<br>TCP window size: 2.50 MByte (default)<br>------------------------------------------------------------<br>[ 3] local 172.23.18.1 port 52325 connected with 172.23.18.10 port 5001<br>[ 4] local 172.23.18.1 port 52326 connected with 172.23.18.10 port 5001<br>[ 5] local 172.23.18.1 port 52327 connected with 172.23.18.10 port 5001<br>[ 6] local 172.23.18.1 port 52328 connected with 172.23.18.10 port 5001<br>[ ID] Interval Transfer Bandwidth<br>[ 4] 0.0-10.0 sec 3.55 GBytes 3.05 Gbits/sec<br>[ 6] 0.0-10.0 sec 3.02 GBytes 2.60 Gbits/sec<br>[ 3] 0.0-10.0 sec 2.91 GBytes 2.50 Gbits/sec<br>[ 5] 0.0-10.0 sec 2.75 GBytes 2.36 Gbits/sec<br>[SUM] 0.0-10.0 sec 12.2 GBytes 10.5 Gbits/sec<br><br>---<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Now, between a storage cluster node and a blade:<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><b>e60-host01 (server):</b><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># iperf -s <br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><b>cibn05 (client):</b><br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"># iperf -c 172.23.18.1 -P 4<br><br>------------------------------------------------------------<br>Client connecting to 172.23.18.1, TCP port 5001<br>TCP window size: 2.50 MByte (default)<br>------------------------------------------------------------<br>[ 6] local 172.23.17.5 port 34263 connected with 172.23.18.1 port 5001<br>[ 4] local 172.23.17.5 port 34260 connected with 172.23.18.1 port 5001<br>[ 5] local 172.23.17.5 port 34262 connected with 172.23.18.1 port 5001<br>[ 3] local 172.23.17.5 port 34261 connected with 172.23.18.1 port 5001<br>[ ID] Interval Transfer Bandwidth<br>[ 4] 0.0- 9.0 sec 3.80 GBytes 3.63 Gbits/sec<br>[ 5] 0.0- 9.0 sec 3.78 GBytes 3.60 Gbits/sec<br>[ 3] 0.0- 9.0 sec 3.78 GBytes 3.61 Gbits/sec<br>[ 6] 0.0-10.0 sec 5.26 GBytes 4.52 Gbits/sec<br>[SUM] 0.0-10.0 sec 16.6 GBytes 14.3 Gbits/sec<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Thanks in advance,<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Best,<br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br clear="all"></div><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"></span></span></div></div></span></div></div></div></div></div>
</div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/users" target="_blank" rel="noreferrer">http://lists.openfabrics.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>