<div dir="ltr">Hi Rosenstock,<div><br></div><div>Thanks for giving quick response.</div><div><br></div><div>We are making the change in configuration as per the input.</div><div><br></div><div>But we are using single IP address for Infiniband  bond0: 192.168.1.1  (IB0 + IB2) Active - Passive Bonding.</div><div><br></div><div>And both the switch are independen as per the attached diagram.</div><div><br></div><div>Is this configuration is valid for <span style="font-family:Calibri,sans-serif;font-size:14.6666669845581px">HCA and Switch level failover requirement.</span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6666669845581px"><br></span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6666669845581px">Thank You</span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6666669845581px">Atul Yadav</span></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 24, 2015 at 6:55 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal@dev.mellanox.co.il" target="_blank">hal@dev.mellanox.co.il</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2/24/2015 7:33 AM, Atul Yadav wrote:<br>
> Hi Rosenstock,<br>
<span class="">><br>
> Thanks for responding.<br>
><br>
> As per our requirement we want to achieve IB bonding for  HCA and Switch<br>
> level fail-over.<br>
><br>
> bond0 (Active- Passive)<br>
> Please provide opensm configuration parameter<br>
<br>
</span>You need to do something along the following lines:<br>
<br>
First, create 2 config files (say opensm-qib0.conf and opensm-qib1.conf)<br>
with the following variables changed as follows:<br>
<br>
opensm-qib0.conf:<br>
# The port GUID on which the OpenSM is running<br>
guid 0x00117500006f5f4c<br>
<br>
# Log file to be used<br>
log_file /var/log/opensm-qib0/opensm.log<br>
<br>
opensm-qib1.conf:<br>
# The port GUID on which the OpenSM is running<br>
guid 0x00117500006f5f4a<br>
<br>
# Log file to be used<br>
log_file /var/log/opensm-qib1/opensm.log<br>
<br>
<br>
then make sure that the following directories exist:<br>
/var/cache/opensm-qib0<br>
/var/log/opensm-qib0<br>
/var/cache/opensm-qib1<br>
/var/log/opensm-qib1<br>
<br>
<br>
and then:<br>
export OSM_CACHE_DIR=/var/cache/opensm-qib0<br>
export OSM_TMP_DIR=/var/log/opensm-qib0<br>
opensm -F opensm-qib0.conf &<br>
<br>
export OSM_CACHE_DIR=/var/cache/opensm-qib1<br>
export OSM_TMP_DIR=/var/log/opensm-qib1<br>
opensm -F opensm-qib1.conf &<br>
<br>
<br>
A similar alternative configuration approach is described in:<br>
<a href="https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg03557.html" target="_blank">https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg03557.html</a><br>
<br>
-- Hal<br>
<div><div class="h5"><br>
><br>
> [root@SRDCB0970RTGMS opensm]# ibstat<br>
> CA 'qib0'<br>
>         CA type: InfiniPath_QLE7342<br>
>         Number of ports: 2<br>
>         Firmware version:<br>
>         Hardware version: 2<br>
>         Node GUID: 0x00117500006f5f4c<br>
>         System image GUID: 0x00117500006f5f4c<br>
>         Port 1:<br>
>                 State: Active<br>
>                 Physical state: LinkUp<br>
>                 Rate: 40<br>
>                 Base lid: 1<br>
>                 LMC: 0<br>
>                 SM lid: 1<br>
>                 Capability mask: 0x0761086a<br>
>                 Port GUID: 0x00117500006f5f4c<br>
>                 Link layer: InfiniBand<br>
>         Port 2:<br>
>                 State: Down<br>
>                 Physical state: Disabled<br>
>                 Rate: 10<br>
>                 Base lid: 65535<br>
>                 LMC: 0<br>
>                 SM lid: 65535<br>
>                 Capability mask: 0x07610868<br>
>                 Port GUID: 0x00117500006f5f4d<br>
>                 Link layer: InfiniBand<br>
> CA 'qib1'<br>
>         CA type: InfiniPath_QLE7342<br>
>         Number of ports: 2<br>
>         Firmware version:<br>
>         Hardware version: 2<br>
>         Node GUID: 0x00117500006f5f4a<br>
>         System image GUID: 0x00117500006f5f4c<br>
>         Port 1:<br>
>                 State: Initializing<br>
>                 Physical state: LinkUp<br>
>                 Rate: 40<br>
>                 Base lid: 65535<br>
>                 LMC: 0<br>
>                 SM lid: 65535<br>
>                 Capability mask: 0x07610868<br>
>                 Port GUID: 0x00117500006f5f4a<br>
>                 Link layer: InfiniBand<br>
>         Port 2:<br>
>                 State: Down<br>
>                 Physical state: Disabled<br>
>                 Rate: 10<br>
>                 Base lid: 65535<br>
>                 LMC: 0<br>
>                 SM lid: 65535<br>
>                 Capability mask: 0x07610868<br>
>                 Port GUID: 0x00117500006f5f4b<br>
>                 Link layer: InfiniBand<br>
><br>
> [root@SRDCB0970RTGMS opensm]#<br>
><br>
> [root@SRDCB0970RTGMS ~]# ibstat -p<br>
> 0x00117500006f5f4c<br>
> 0x00117500006f5f4d<br>
> 0x00117500006f5f4a<br>
><br>
> 0x00117500006f5f4b<br>
><br>
><br>
> [root@SRDCB0970RTGMS ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0<br>
> DEVICE=bond0<br>
> IPADDR=192.168.1.1<br>
> NETMASK=<a href="tel:255.255.255.0" value="+912552552550">255.255.255.0</a><br>
> BROADCAST=<a href="tel:192.168.1.255" value="+911921681255">192.168.1.255</a><br>
> ONBOOT=yes<br>
> BOOTPROTO=none<br>
> USERCTL=no<br>
> MTU=65520<br>
> BONDING_OPTS=" mode=1 primary=ib0 updelay=0 downdelay=0"<br>
> [root@SRDCB0970RTGMS ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib*<br>
> DEVICE=ib0<br>
> USERCTL=no<br>
> ONBOOT=yes<br>
> MASTER=bond0<br>
> SLAVE=yes<br>
> BOOTPROTO=none<br>
> TYPE=Infiniband<br>
> PRIMARY=yes<br>
> DEVICE=ib1<br>
> USERCTL=no<br>
> ONBOOT=yes<br>
> MASTER=bond0<br>
> SLAVE=yes<br>
> BOOTPROTO=none<br>
> TYPE=Infiniband<br>
> DEVICE=ib2<br>
> USERCTL=no<br>
> ONBOOT=yes<br>
> MASTER=bond0<br>
> SLAVE=yes<br>
> BOOTPROTO=none<br>
> TYPE=Infiniband<br>
> DEVICE=ib3<br>
> USERCTL=no<br>
> ONBOOT=yes<br>
> MASTER=bond0<br>
> SLAVE=yes<br>
> BOOTPROTO=none<br>
> TYPE=Infiniband<br>
> [root@SRDCB0970RTGMS ~]#<br>
><br>
><br>
><br>
> Thank You<br>
><br>
> Atul Yadav<br>
><br>
><br>
><br>
><br>
><br>
> On Tue, Feb 24, 2015 at 5:55 PM, Hal Rosenstock <<a href="mailto:hal@dev.mellanox.co.il">hal@dev.mellanox.co.il</a><br>
</div></div><span class="">> <mailto:<a href="mailto:hal@dev.mellanox.co.il">hal@dev.mellanox.co.il</a>>> wrote:<br>
><br>
>     On 2/24/2015 5:19 AM, Atul Yadav wrote:<br>
>     > Hi Team,<br>
>     ><br>
>     > We are trying to setup the HCA and Switch level failover.<br>
>     ><br>
>     > Operating System:- Centos 6.5<br>
>     ><br>
>     > Please guide us<br>
><br>
>     To run OpenSM on multiple ports/HCAs on the same machine for the same<br>
>     subnet, multiple instances of OpenSM need to be invoked one per port/HCA<br>
>     and need separate but similar configuration.<br>
><br>
>     -- Hal<br>
><br>
>     ><br>
>     > Thank You<br>
>     ><br>
>     > Atul Yadav<br>
>     ><br>
>     ><br>
>     ><br>
>     > _______________________________________________<br>
>     > ewg mailing list<br>
</span>>     > <a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a> <mailto:<a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a>><br>
>     > <a href="http://lists.openfabrics.org/mailman/listinfo/ewg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/ewg</a><br>
><br>
><br>
<br>
</blockquote></div><br></div>