<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Hi All,<br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">I'm having some issues with my IB network, basically I've two SX6036G as Spine switches, two SX6036F as Leaf switches each connected to both Spines, SM is running on one of the Spines. Then I've connected some supermicro servers to the Leaf switches, create a BOND active-passive, ping response fine, but, when I try to run a ib_write_bw test I'm getting:<br><br><br>Server:<br><br># ib_write_bw<br><br>************************************<br>* Waiting for client to connect... *<br>************************************<br>---------------------------------------------------------------------------------------<br> RDMA_Write BW Test<br> Dual-port : OFF Device : mlx4_0<br> Number of qps : 1 Transport type : IB<br> Connection type : RC Using SRQ : OFF<br> CQ Moderation : 100<br> Mtu : 2048[B]<br> Link type : IB<br> Max inline data : 0[B]<br> rdma_cm QPs : OFF<br> Data ex. method : Ethernet<br>---------------------------------------------------------------------------------------<br> local address: LID 0x63 QPN 0x0296 PSN 0xa23c53 RKey 0xb8011100 VAddr 0x007fc3c655c000<br> remote address: LID 0x03 QPN 0x0219 PSN 0xc0ef7c RKey 0x8011100 VAddr 0x007f62cc318000<br>---------------------------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]<br>ethernet_read_keys: Couldn't read remote address<br> Unable to read to socket/rdam_cm<br> Failed to exchange data between server and clients<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Client:<br><br># ib_write_bw 172.23.16.1<br>---------------------------------------------------------------------------------------<br> RDMA_Write BW Test<br> Dual-port : OFF Device : mlx4_1<br> Number of qps : 1 Transport type : IB<br> Connection type : RC Using SRQ : OFF<br> TX depth : 128<br> CQ Moderation : 100<br> Mtu : 2048[B]<br> Link type : IB<br> Max inline data : 0[B]<br> rdma_cm QPs : OFF<br> Data ex. method : Ethernet<br>---------------------------------------------------------------------------------------<br> local address: LID 0x03 QPN 0x0219 PSN 0xc0ef7c RKey 0x8011100 VAddr 0x007f62cc318000<br> remote address: LID 0x63 QPN 0x0296 PSN 0xa23c53 RKey 0xb8011100 VAddr 0x007fc3c655c000<br>---------------------------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]<br>Problems with warm up<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">the bond configuration is the following:<br><br>auto ib0<br>iface ib0 inet manual<br> bond-master bond-PUB<br> bond-primary ib0<br> pre-up echo connected > /sys/class/net/ib0/mode<br><br>auto ib1<br>iface ib1 inet manual<br> bond-master bond-PUB<br> pre-up echo connected > /sys/class/net/ib1/mode<br><br>auto bond-PUB<br>iface bond-PUB inet static<br> address 172.23.16.1<br> netmask 255.255.240.0<br> network 172.23.16.0<br> bond-mode active-backup<br> bond-miimon 100<br> bond-slaves none<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">AND<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">auto ib1<br>iface ib1 inet manual<br> bond-master bond-PUB<br> bond-primary ib1<br> pre-up echo connected > /sys/class/net/ib1/mode<br><br>auto ib3<br>iface ib3 inet manual<br> bond-master bond-PUB<br> pre-up echo connected > /sys/class/net/ib3/mode<br><br>auto bond-PUB<br>iface bond-PUB inet static<br> address 172.23.17.4<br> netmask 255.255.240.0<br> network 172.23.16.0<br> bond-mode active-backup<br> bond-miimon 100<br> bond-slaves none<br><br>## <br>auto ib0<br>iface ib0 inet manual<br> bond-master bond-CLUS<br> bond-primary ib0<br> pre-up echo connected > /sys/class/net/ib0/mode<br><br>auto ib2<br>iface ib2 inet manual<br> bond-master bond-CLUS<br> pre-up echo connected > /sys/class/net/ib2/mode<br><br>auto bond-CLUS<br>iface bond-CLUS inet static<br> address 172.23.32.4<br> netmask 255.255.240.0<br> network 172.23.32.0<br> bond-mode active-backup<br> bond-miimon 100<br> bond-slaves none<br><br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Any ideas? Also the performance of a FIO operation between hosts is REALLY slow.. on the range of 100-150MB/s...with different operations and bs.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Thanks in advance,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)">Best,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small;color:rgb(11,83,148)"><br clear="all"></div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="font-size:10pt;font-family:Trebuchet MS,sans-serif;color:rgb(68,68,68)"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"></span></span></div></div></span></div></div></div></div></div>
</div>