<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Hi all,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">I'm having some issues with my IB network, basically I have the following setup (pdf attach). I run a fio test between the HP Blade with QDR (bonding ports active/backup mode), to a storage cluster with FDR (with no bonding at all), and the best result that I can get is 1.7GB/s, that's pretty slow actually. However I was hopping something between 2.5-3.5GB/s on a QDR infiniband network. Then I try to tweak some parameters, for example setting the scaling_governor to 'performance', and set the 'connected' mode in the ib ports, then change the following variables:<br><br>sysctl -w net.core.netdev_max_backlog=250000<br>sysctl -w net.core.rmem_max=4194304<br>sysctl -w net.core.wmem_max=4194304<br>sysctl -w net.core.rmem_default=4194304<br>sysctl -w net.core.wmem_default=4194304<br>sysctl -w net.core.optmem_max=4194304<br>sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"<br>sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304"<br>sysctl -w net.ipv4.tcp_low_latency=1<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">The bond configuration is the following:<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># cat /etc/modprobe.d/bonding.conf<br><br>alias bond-ib bonding options bonding mode=1 miimon=100 downdelay=100 updelay=100 max_bonds=2<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># cat /etc/network/interfaces<br><br>(...)<br><br>## INFINIBAND CONF<br>auto ib0<br>iface ib0 inet manual<br> bond-master bond-ib<br><br>auto ib1<br>iface ib1 inet manual<br> bond-master bond-ib<br><br>auto bond-ib<br>iface bond-ib inet static<br> address 172.23.18.1<br> netmask 255.255.240.0<br> slaves ib0 ib1<br> bond_miimon 100<br> bond_mode active-backup<br> pre-up echo connected > /sys/class/net/ib0/mode<br> pre-up echo connected > /sys/class/net/ib1/mode<br> pre-up /sbin/ifconfig ib0 mtu 65520<br> pre-up /sbin/ifconfig ib1 mtu 65520<br> pre-up modprobe bond-ib<br> pre-up /sbin/ifconfig bond-ib mtu 65520<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">OS is Ubuntu 14.04.3 LTS on the HP blade with Kernel 3.13.0-63-generic, and Ubuntu 14.04.3 LTS with kernel 3.19.0-25-generic for the storage cluster.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">The IB Mezzanine cards on the HP Blades are "InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)". And on the storage cluster the IB ADPT are "Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]<br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Then I run in one of the nodes in cluster the '<b>ibqueryerrors</b>' command and found the following:<br><br>$ ibqueryerrors<br>Errors for "e60-host01 HCA-1" <span style="color:rgb(255,0,0)">---> blade1 the one with the bonding configuration using internally HP-IB-SW port 17 and 25</span><br> GUID 0xf452140300dd3296 port 2: [PortXmitWait == 15]<br>Errors for 0x2c902004b0918 "Infiniscale-IV Mellanox Technologies"<br> GUID 0x2c902004b0918 port ALL: [PortXmitWait == 325727936]<br> GUID 0x2c902004b0918 port 25: [PortXmitWait == 325727936]<br>Errors for 0xe41d2d030031e9c1 "MF0;GWIB01:SX6036G/U1"<br> GUID 0xe41d2d030031e9c1 port ALL: [PortXmitWait == 326981305]<br> GUID 0xe41d2d030031e9c1 port 11: [PortXmitWait == 326976642]<br> GUID 0xe41d2d030031e9c1 port 36: [PortXmitWait == 4663]<br>Errors for 0xf45214030073f500 "MF0;SWIB02:SX6018/U1"<br> GUID 0xf45214030073f500 port ALL: [PortXmitWait == 13979524]<br> GUID 0xf45214030073f500 port 8: [PortXmitWait == 3749467]<br> GUID 0xf45214030073f500 port 9: [PortXmitWait == 3434343]<br> GUID 0xf45214030073f500 port 10: [PortXmitWait == 3389114]<br> GUID 0xf45214030073f500 port 11: [PortXmitWait == 3406600]<br>Errors for 0xe41d2d030031eb41 "MF0;GWIB02:SX6036G/U1"<br> GUID 0xe41d2d030031eb41 port ALL: [PortXmitWait == 1352]<br> GUID 0xe41d2d030031eb41 port 34: [PortXmitWait == 1352]<br>Errors for "cibn08 HCA-1"<br> GUID 0xe41d2d03007b77c1 port 1: [PortXmitWait == 813152781]<br> GUID 0xe41d2d03007b77c2 port 2: [PortXmitWait == 3256286]<br>Errors for "cibn07 HCA-1"<br> GUID 0xe41d2d03007b67c1 port 1: [PortXmitWait == 841850209]<br> GUID 0xe41d2d03007b67c2 port 2: [PortXmitWait == 3211488]<br>Errors for "cibn05 HCA-1"<br> GUID 0xe41d2d0300d95191 port 1: [PortXmitWait == 840576923]<br> GUID 0xe41d2d0300d95192 port 2: [PortXmitWait == 2635901]<br>Errors for "cibn06 HCA-1"<br> GUID 0xe41d2d03007b77b1 port 1: [PortXmitWait == 843231930]<br> GUID 0xe41d2d03007b77b2 port 2: [PortXmitWait == 2869022]<br>Errors for 0xe41d2d0300097630 "MF0;SWIB01:SX6018/U1"<br> GUID 0xe41d2d0300097630 port ALL: [PortXmitWait == 470746689]<br> GUID 0xe41d2d0300097630 port 0: [PortXmitWait == 7]<br> GUID 0xe41d2d0300097630 port 2: [PortXmitWait == 8046]<br> GUID 0xe41d2d0300097630 port 3: [PortXmitWait == 7631]<br> GUID 0xe41d2d0300097630 port 8: [PortXmitWait == 219608]<br> GUID 0xe41d2d0300097630 port 9: [PortXmitWait == 216118]<br> GUID 0xe41d2d0300097630 port 10: [PortXmitWait == 198693]<br> GUID 0xe41d2d0300097630 port 11: [PortXmitWait == 206192]<br> GUID 0xe41d2d0300097630 port 18: [PortXmitWait == 469890394]<br>Errors for "cibm01 HCA-1"<br> GUID 0xe41d2d0300163651 port 1: [PortXmitWait == 6002]<br><br>## Summary: 22 nodes checked, 11 bad nodes found<br>## 208 ports checked, 26 ports have errors beyond threshold<br>## <br>## Suppressed:<br><br><br>$ ibportstate -L 29 17 query<br>Switch PortInfo:<br># Port info: Lid 29 port 17<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................75<br>SMLid:...........................2328<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Peer PortInfo:<br># Port info: Lid 29 DR path slid 4; dlid 65535; 0,17 port 1<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................32<br>SMLid:...........................2<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Mkey:............................<not displayed><br>MkeyLeasePeriod:.................0<br>ProtectBits:.....................0<br><br>---<br><br>$ ibportstate -L 29 25 query<br>Switch PortInfo:<br># Port info: Lid 29 port 25<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................75<br>SMLid:...........................2328<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Peer PortInfo:<br># Port info: Lid 29 DR path slid 4; dlid 65535; 0,25 port 2<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................33<br>SMLid:...........................2<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br>Mkey:............................<not displayed><br>MkeyLeasePeriod:.................0<br>ProtectBits:.....................0<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">First I thought that maybe some cables could be in a bad state, but.. all of them?... so I really don't know if maybe this XmitWait could be pushing some noise on the performance at all or not. Any ideas? or hints? Also I had the SM configured on SWIB01 with high priority and then a second SM configured on SWIB02 with less priority, both in an active state, is this ok? Or is better to only have one and only one SM active at a time in the entire IB network?<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Also find below some iperf tests between blades that are on different enclosures:<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><b>e61-host01 (server):</b><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># iperf -s<br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br><b>e60-host01 (client):</b><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># iperf -c 172.23.18.10 -P 4<br><br>------------------------------------------------------------<br>Client connecting to 172.23.18.10, TCP port 5001<br>TCP window size: 2.50 MByte (default)<br>------------------------------------------------------------<br>[ 3] local 172.23.18.1 port 52325 connected with 172.23.18.10 port 5001<br>[ 4] local 172.23.18.1 port 52326 connected with 172.23.18.10 port 5001<br>[ 5] local 172.23.18.1 port 52327 connected with 172.23.18.10 port 5001<br>[ 6] local 172.23.18.1 port 52328 connected with 172.23.18.10 port 5001<br>[ ID] Interval Transfer Bandwidth<br>[ 4] 0.0-10.0 sec 3.55 GBytes 3.05 Gbits/sec<br>[ 6] 0.0-10.0 sec 3.02 GBytes 2.60 Gbits/sec<br>[ 3] 0.0-10.0 sec 2.91 GBytes 2.50 Gbits/sec<br>[ 5] 0.0-10.0 sec 2.75 GBytes 2.36 Gbits/sec<br>[SUM] 0.0-10.0 sec 12.2 GBytes 10.5 Gbits/sec<br><br>---<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Now, between a storage cluster node and a blade:<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><b>e60-host01 (server):</b><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># iperf -s <br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><b>cibn05 (client):</b><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"># iperf -c 172.23.18.1 -P 4<br><br>------------------------------------------------------------<br>Client connecting to 172.23.18.1, TCP port 5001<br>TCP window size: 2.50 MByte (default)<br>------------------------------------------------------------<br>[ 6] local 172.23.17.5 port 34263 connected with 172.23.18.1 port 5001<br>[ 4] local 172.23.17.5 port 34260 connected with 172.23.18.1 port 5001<br>[ 5] local 172.23.17.5 port 34262 connected with 172.23.18.1 port 5001<br>[ 3] local 172.23.17.5 port 34261 connected with 172.23.18.1 port 5001<br>[ ID] Interval Transfer Bandwidth<br>[ 4] 0.0- 9.0 sec 3.80 GBytes 3.63 Gbits/sec<br>[ 5] 0.0- 9.0 sec 3.78 GBytes 3.60 Gbits/sec<br>[ 3] 0.0- 9.0 sec 3.78 GBytes 3.61 Gbits/sec<br>[ 6] 0.0-10.0 sec 5.26 GBytes 4.52 Gbits/sec<br>[SUM] 0.0-10.0 sec 16.6 GBytes 14.3 Gbits/sec<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Thanks in advance,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Best,<br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br clear="all"></div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="font-size:10pt;font-family:Trebuchet MS,sans-serif;color:rgb(68,68,68)"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"></span></span></div></div></span></div></div></div></div></div>
</div>