<div dir="ltr">But, I'm trying to route from one HCA port to another HCA port (not a switch). I'm taking down a switch link in which there is another path available between the HCA ports. Do the port GUIDs change in this type of event (I don't believe that is the case).<div>
<br></div><div>When I take this switch port down I would expect the output to be:</div><div><div>From {0x0002c90300ebbb60}[2]</div><div>[2] -> {0x0002c90300684e30}[19]</div><div><b>[1] -> {0x0002c90200431fb8}[10]</b></div>
<div><b>[33(or 34)] -> {0x001397010a000044}[8(or 9)]</b></div><div>[35] -> {0x0013970301001f4c}[1]</div><div>To {0x0013970301001f4b}[1]</div></div><div><br></div><div>I understand if I disconnect the HCA port then I should not be able to connect, but taking down a switch port should cause ibsim/opensm to reroute around the downed link. Again, please let me know if I'm missing something because I'm still learning this.</div>
<div><br></div><div>Thank,</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Brigham Young University</span></div>
<br><br><div class="gmail_quote">On Fri, Oct 25, 2013 at 12:35 PM, Albert Chu <span dir="ltr"><<a href="mailto:chu11@llnl.gov" target="_blank">chu11@llnl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Robert,<br>
<div class="im"><br>
> I'm trying to test routing in ibsim, but it doesn't seem to update the<br>
> routing tables in the simulated switches. If I take a link down using<br>
> the clear command in ibsim, I see opensm saying that it is updating<br>
> the routing tables and that it completes, but I can't ibtracert to the<br>
> LID who's path was taken down.<br>
<br>
</div>I have a feeling you might be confusing ibtracert's behavior w/ the<br>
typical behavior of traceroute.<br>
<br>
When you disable the link below, you are effectively taking node(s) out<br>
of your fabric. OpenSM will see that the node(s) disappeared and will<br>
re-route the fabric. Those nodes are now eliminated from all of the<br>
routing tables. So when you ibtracert that node, ibtracert effectively<br>
states it can't do a traceroute b/c the node/route doesn't exist.<br>
<br>
This is different than traceroute, which output the network hops as far<br>
as it can go, even if the end destination is down.<br>
<br>
Al<br>
<div class="HOEnZb"><div class="h5"><br>
On Fri, 2013-10-25 at 12:22 -0600, Robert LeBlanc wrote:<br>
> I just realized that in this example I'm shutting down the entire<br>
> switch that the host is connected to instead of the uplink port. If I<br>
> issue 'clear "S-0002c90300684e30" 2"', I get the same result. Port 1<br>
> and 2 are both uplink ports to different leaf IB switches in a fat<br>
> tree scheme.<br>
><br>
><br>
><br>
> Robert LeBlanc<br>
> OIT Infrastructure & Virtualization Engineer<br>
> Brigham Young University<br>
><br>
><br>
> On Fri, Oct 25, 2013 at 11:19 AM, Robert LeBlanc<br>
> <<a href="mailto:robert_leblanc@byu.edu">robert_leblanc@byu.edu</a>> wrote:<br>
> Here is the details of what I'm doing:<br>
><br>
><br>
> In one terminal, I run ibsim:<br>
> root@rleblanc-pc:/home/leblanc/Downloads# ibsim -s ibtopo<br>
> parsing: ibtopo<br>
> ibtopo: parsed 928 lines<br>
> ########################<br>
> Network simulator ready.<br>
> MaxNetNodes = 2048<br>
> MaxNetSwitches = 256<br>
> MaxNetPorts = 13312<br>
> MaxLinearCap = 30720<br>
> MaxMcastCap = 1024<br>
> sim> ibwarn: [2278] process_packet: no one to handle pkt:<br>
> class 0x81, attr 0xff90<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> ...snip out tons of these messages...<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> clear "S-0002c90300684e30"<br>
> sim> ibwarn: [2278] process_packet: got trap repress - drop<br>
> ibwarn: [2278] process_packet: got trap repress - drop<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> ...snip out tons of these messages...<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> relink "0002c90300684e30"<br>
> # nodeid "0002c90300684e30" (0002c90300684e30) not found<br>
> sim> relink "S-0002c90300684e30"<br>
> sim> ibwarn: [2278] process_packet: got trap repress - drop<br>
> ibwarn: [2278] process_packet: got trap repress - drop<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> ...snip out tons of these messages...<br>
> ibwarn: [2278] process_packet: no one to handle pkt: class<br>
> 0x81, attr 0xff90<br>
> quit<br>
> Exiting network simulator.<br>
> root@rleblanc-pc:/home/leblanc/Downloads#<br>
><br>
><br>
> Then in another terminal I run opensm:<br>
> root@rleblanc-pc:/home/leblanc/Documents/Work/Scripts/ib#<br>
> SIM_HOST="H-0013970201000978" OSM_TMP_DIR=./ OSM_CACHE_DIR=./<br>
> LD_PRELOAD=/usr/lib/umad2sim/libumad2sim.so opensm -e -v<br>
> -f ./osm.log<br>
> -------------------------------------------------<br>
> OpenSM 3.3.15<br>
> Command Line Arguments:<br>
> Creating new log file<br>
> Verbose option -v (log flags = 0x7)<br>
> Log File: ./osm.log<br>
> -------------------------------------------------<br>
> OpenSM 3.3.15<br>
><br>
><br>
> Entering DISCOVERING state<br>
><br>
><br>
> Using default GUID 0x13970201000979<br>
> Entering MASTER state<br>
><br>
><br>
><br>
><br>
> =======================================================================================================<br>
> Vendor : Ty : # : Sta : LID : LMC : MTU : LWA : LSA :<br>
> Port GUID : Neighbor Port (Port #)<br>
> Unknown : CA : 01 : ACT : 0003 : 0 : 2048 : 4x : 2.5 :<br>
> f04da29097793001 : 0002c9020042ea60 (12)<br>
> Unknown : CA : 02 : ACT : 0007 : 0 : 2048 : 4x : 2.5 :<br>
> f04da29097793002 : 0002c902004294e0 (12)<br>
> ------------------------------------------------------------------------------------------------------<br>
> Mellanox : SW : 00 : : 0002 : 0 : : : :<br>
> 0002c90300879a00 :<br>
> Mellanox : SW : 01 : ACT : : : 2048 : 4x : 2.5 :<br>
> 0002c90300879a00 : 0002c90200431f90 (08)<br>
> Mellanox : SW : 02 : ACT : : : 2048 : 4x : 2.5 :<br>
> 0002c90300879a00 : 0002c90200431f58 (09)<br>
> Mellanox : SW : 03 : DWN : : : ??? : ??? : Ext :<br>
> 0002c90300879a00 :<br>
> ...snip...<br>
><br>
><br>
> Then in a third console I run ibtracert:<br>
> leblanc@rleblanc-pc:~/Documents/Work/Scripts/ib$<br>
> LD_PRELOAD=/usr/lib/umad2sim/libumad2sim.so /usr/sbin/ibtracert -G -n 0x0002c90300ebbb62 0x0013970301001f4c 2> /dev/null<br>
> From {0x0002c90300ebbb60}[2]<br>
> [2] -> {0x0002c90300684e30}[19]<br>
> [2] -> {0x0002c90200431eb8}[10]<br>
> [33] -> {0x001397010a000044}[10]<br>
> [35] -> {0x0013970301001f4c}[1]<br>
> To {0x0013970301001f4b}[1]<br>
> leblanc@rleblanc-pc:~/Documents/Work/Scripts/ib$<br>
> LD_PRELOAD=/usr/lib/umad2sim/libumad2sim.so /usr/sbin/ibtracert -G -n 0x0002c90300ebbb62 0x0013970301001f4c 2> /dev/null<br>
> /usr/sbin/ibtracert: iberror: failed: can't resolve source<br>
> port 0x0002c90300ebbb62<br>
> leblanc@rleblanc-pc:~/Documents/Work/Scripts/ib$<br>
> LD_PRELOAD=/usr/lib/umad2sim/libumad2sim.so /usr/sbin/ibtracert -G -n 0x0002c90300ebbb62 0x0013970301001f4c 2> /dev/null<br>
> From {0x0002c90300ebbb60}[2]<br>
> [2] -> {0x0002c90300684e30}[19]<br>
> [2] -> {0x0002c90200431eb8}[10]<br>
> [33] -> {0x001397010a000044}[10]<br>
> [35] -> {0x0013970301001f4c}[1]<br>
> To {0x0013970301001f4b}[1]<br>
> leblanc@rleblanc-pc:~/Documents/Work/Scripts/ib$<br>
><br>
><br>
> I'm attaching our topo file that we are using and the opensm<br>
> logs (you should be able to replicate the problem given this<br>
> information or tell me what I'm doing wrong).<br>
><br>
><br>
> Thanks,<br>
><br>
><br>
><br>
> Robert LeBlanc<br>
> OIT Infrastructure & Virtualization Engineer<br>
> Brigham Young University<br>
><br>
><br>
><br>
> On Tue, Oct 22, 2013 at 10:55 PM, Hal Rosenstock<br>
> <<a href="mailto:hal.rosenstock@gmail.com">hal.rosenstock@gmail.com</a>> wrote:<br>
> ibsim just simulates the network (topology, SMAs, and<br>
> PMAs). OpenSM configured the subnet including the<br>
> routing (LFTs and MFTs) based on the routing<br>
> algorithm. It is possible in a topology that multiple<br>
> routing algorithms yield the same routes. More<br>
> specifics would be needed to comment "deeper"...<br>
><br>
> -- Hal<br>
><br>
><br>
> On Tue, Oct 22, 2013 at 6:38 PM, Robert LeBlanc<br>
> <<a href="mailto:robert_leblanc@byu.edu">robert_leblanc@byu.edu</a>> wrote:<br>
><br>
> I'm trying to test routing in ibsim, but it<br>
> doesn't seem to update the routing tables in<br>
> the simulated switches. If I take a link down<br>
> using the clear command in ibsim, I see opensm<br>
> saying that it is updating the routing tables<br>
> and that it completes, but I can't ibtracert<br>
> to the LID who's path was taken down.<br>
><br>
><br>
> Should ibsim and opensm be reconfiguring<br>
> routing in the simulated environment? No<br>
> matter which routing protocol I select in<br>
> opensm, the routes are always the same, even<br>
> having opensm re-LID the entire fabric doesn't<br>
> help. Any help would be appreciated.<br>
><br>
><br>
> Output from opensm:<br>
><br>
><br>
> ******************************************************************<br>
> ***** LID ASSIGNMENT COMPLETE - STARTING<br>
> SWITCH TABLE CONFIG *****<br>
> ******************************************************************<br>
><br>
><br>
><br>
><br>
> Oct 22 16:27:20 330198 [8437A700] 0x04 -><br>
> osm_ucast_mgr_build_lid_matrices: Starting<br>
> switches' Min Hop Table Assignment<br>
> Oct 22 16:27:20 330954 [8437A700] 0x02 -><br>
> osm_ucast_mgr_process: minhop tables<br>
> configured on all switches<br>
> Oct 22 16:27:20 331191 [8437A700] 0x04 -><br>
> do_sweep:<br>
><br>
><br>
><br>
><br>
> ******************************************************************<br>
> **************** SWITCHES CONFIGURED FOR<br>
> UNICAST *****************<br>
> ******************************************************************<br>
><br>
><br>
><br>
><br>
> Thanks,<br>
><br>
><br>
> Robert LeBlanc<br>
> OIT Infrastructure & Virtualization Engineer<br>
> Brigham Young University<br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.openfabrics.org">Users@lists.openfabrics.org</a><br>
> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.openfabrics.org">Users@lists.openfabrics.org</a><br>
> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Albert Chu<br>
<a href="mailto:chu11@llnl.gov">chu11@llnl.gov</a><br>
Computer Scientist<br>
High Performance Systems Division<br>
Lawrence Livermore National Laboratory<br>
<br>
<br>
</font></span></blockquote></div><br></div>