<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] SM Bad Port Handling</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Hi Shahar,</FONT>
</P>
<P><FONT SIZE=2>> Your analysis is not completely accurate. The SM configure the</FONT>
<BR><FONT SIZE=2>> subnet using direct mads only, and it builds a spanning tree of direct</FONT>
<BR><FONT SIZE=2>> routes. What I want to say, is that that it doesn't matter why exactly a</FONT>
<BR><FONT SIZE=2>> port is unreachable. Once a port can not be reached, you can either</FONT>
<BR><FONT SIZE=2>> retry the entire heavy sweep process, but if the problem repeats itself</FONT>
<BR><FONT SIZE=2>> (X times) on the same port, you have no alternative other then disable</FONT>
<BR><FONT SIZE=2>> it.</FONT>
<BR><FONT SIZE=2>The point is that the real "bad" ports are not the ones that are killing 100% of packets</FONT>
<BR><FONT SIZE=2>(since they will simply have a "DOWN" state and vanish).</FONT>
</P>
<P><FONT SIZE=2>The real bad ports are the ones that pass < 25% (as we use retry of 4) of packets that goes through them. </FONT>
</P>
<P><FONT SIZE=2>When such a port happen to be on a switch it will normally cause other ports to appear to be "bad" - NOT ITSELF !</FONT>
<BR><FONT SIZE=2>The reason for it is that the number of packets sent through a switch port (not a leaf switch port) is much larger then the number of packets that deals with the discovery of the port itself. All the ports "behind" the switch port will go through that port. And there is a much higher chance for ALL the packets that goes to an end-port be dropped then the chance for ALL the packets that goes through the switch ports to be dropped).</FONT></P>
<P><FONT SIZE=2>So if you implement the feature the way it was proposed what you will end up with is disconnecting end-ports and not the real bad port.</FONT></P>
<P><FONT SIZE=2>Why is it bad? It is bad since in tree topology the end-ports always have an alternate path to the SM. If you could find the real flaky bad port - you could still communicate with all the end-ports.</FONT></P>
<P><FONT SIZE=2>So how do we find that bad port/cable that causes other port to appear bad?</FONT>
<BR><FONT SIZE=2>We have internally had many long discussions on this topic. The algorithm is not fully developed yet. But several things are clear:</FONT></P>
<P><FONT SIZE=2>1. One needs to track the number of successful and bad packet flowing through each port. Such that a failure rate can be obtained for each port.</FONT></P>
<P><FONT SIZE=2>2. Topology based analysis should be used to find the common point that is first to have a high drop rate on the directed route tree.</FONT></P>
<P><FONT SIZE=2>3. Alternate directed routes might be used to invalidate "suspicious" ports.</FONT>
</P>
<P><FONT SIZE=2>In any case, I was not proposing relying on traps. I was suggesting to use the </FONT>
<BR><FONT SIZE=2>"healthy" bit on physical ports as the way to carry the information about "bad" ports (once we correctly find them) into the rest of the algorithms used by the SM.</FONT></P>
<P><FONT SIZE=2>Regarding the need to "disconnect" a bad HCA "end-port" - I still have not seen any log showing OpenSM going through infinite "polling" of bad ports. As I know the code - I can not believe this is possible - so unless you have a log that shows this phenomena (and not another one) please do not chance this path.</FONT></P>
<P><FONT SIZE=2>One last word. I would highly recommend using the management simulator for setting arbitrary (random) bad packet drops and test any algorithm you might think of.</FONT></P>
<P><FONT SIZE=2>EZ</FONT>
</P>
<P><FONT SIZE=2>Eitan Zahavi</FONT>
<BR><FONT SIZE=2>Design Technology Director</FONT>
<BR><FONT SIZE=2>Mellanox Technologies LTD</FONT>
<BR><FONT SIZE=2>Tel:+972-4-9097208</FONT>
<BR><FONT SIZE=2>Fax:+972-4-9593245</FONT>
<BR><FONT SIZE=2>P.O. Box 586 Yokneam 20692 ISRAEL</FONT>
</P>
<BR>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: shaharf [<A HREF="mailto:shaharf@voltaire.com">mailto:shaharf@voltaire.com</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Wednesday, April 13, 2005 5:03 PM</FONT>
<BR><FONT SIZE=2>> To: Eitan Zahavi; Hal Rosenstock</FONT>
<BR><FONT SIZE=2>> Cc: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>> Subject: RE: [openib-general] SM Bad Port Handling</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Eitan,</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Your analysis is not completely accurate. The SM configure the</FONT>
<BR><FONT SIZE=2>> subnet using direct mads only, and it builds a spanning tree of direct</FONT>
<BR><FONT SIZE=2>> routes. What I want to say, is that that it doesn't matter why exactly a</FONT>
<BR><FONT SIZE=2>> port is unreachable. Once a port can not be reached, you can either</FONT>
<BR><FONT SIZE=2>> retry the entire heavy sweep process, but if the problem repeats itself</FONT>
<BR><FONT SIZE=2>> (X times) on the same port, you have no alternative other then disable</FONT>
<BR><FONT SIZE=2>> it. If the SM will have an alternative method of building direct paths,</FONT>
<BR><FONT SIZE=2>> then such alternative path could be attempted. Currently it is not</FONT>
<BR><FONT SIZE=2>> relevant. Speaking of "statistical analysis", what are the odds that a</FONT>
<BR><FONT SIZE=2>> port will behave well when it is queried directly, but starts to loose</FONT>
<BR><FONT SIZE=2>> packets when a direct route is routed through it, and behave</FONT>
<BR><FONT SIZE=2>> consistently during all retries? Again, even if this is the case (and in</FONT>
<BR><FONT SIZE=2>> understatement, I am not sure how frequent it is), the port behind it is</FONT>
<BR><FONT SIZE=2>> unreachable and therefore "bad".</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The current unhealthy port mechanism is not redundant to this "bad" port</FONT>
<BR><FONT SIZE=2>> mechanism because it does not handle the same case. Both mechanisms are</FONT>
<BR><FONT SIZE=2>> required. The issue if they can share the same status bit is really an</FONT>
<BR><FONT SIZE=2>> implementation issue.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Relying of traps is very problematic in some cases, particularly in</FONT>
<BR><FONT SIZE=2>> initial bring up sweep when the SM lid is not even configured (remember</FONT>
<BR><FONT SIZE=2>> VTEC?).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Shahar</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ________________________________________</FONT>
<BR><FONT SIZE=2>> From: openib-general-bounces@openib.org</FONT>
<BR><FONT SIZE=2>> [<A HREF="mailto:openib-general-bounces@openib.org">mailto:openib-general-bounces@openib.org</A>] On Behalf Of Eitan Zahavi</FONT>
<BR><FONT SIZE=2>> Sent: Wednesday, April 13, 2005 11:21 AM</FONT>
<BR><FONT SIZE=2>> To: Hal Rosenstock; Eitan Zahavi</FONT>
<BR><FONT SIZE=2>> Cc: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>> Subject: RE: [openib-general] SM Bad Port Handling</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I probably did not make point very clear:</FONT>
<BR><FONT SIZE=2>> It is bad (not to say wrong) to disqualify a port and mark it as bad</FONT>
<BR><FONT SIZE=2>> port if it did not respond to queries.</FONT>
<BR><FONT SIZE=2>> The cause of the issue might be a flaky link on the directed route to</FONT>
<BR><FONT SIZE=2>> the port.</FONT>
<BR><FONT SIZE=2>> If the SM would be able to find that flaky link port it would avoid</FONT>
<BR><FONT SIZE=2>> marking the wrong ports. More over, the port that was almost marked as</FONT>
<BR><FONT SIZE=2>> bad by the simplistic algorithm you propose will be discovered and</FONT>
<BR><FONT SIZE=2>> operational as there many other paths to reach it - walking around the</FONT>
<BR><FONT SIZE=2>> real bad port !</FONT>
<BR><FONT SIZE=2>> Eitan Zahavi</FONT>
<BR><FONT SIZE=2>> Design Technology Director</FONT>
<BR><FONT SIZE=2>> Mellanox Technologies LTD</FONT>
<BR><FONT SIZE=2>> Tel:+972-4-9097208</FONT>
<BR><FONT SIZE=2>> Fax:+972-4-9593245</FONT>
<BR><FONT SIZE=2>> P.O. Box 586 Yokneam 20692 ISRAEL</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> > -----Original Message-----</FONT>
<BR><FONT SIZE=2>> > From: Hal Rosenstock [<A HREF="mailto:halr@voltaire.com">mailto:halr@voltaire.com</A>]</FONT>
<BR><FONT SIZE=2>> > Sent: Wednesday, April 13, 2005 12:00 PM</FONT>
<BR><FONT SIZE=2>> > To: Eitan Zahavi</FONT>
<BR><FONT SIZE=2>> > Cc: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>> > Subject: RE: [openib-general] SM Bad Port Handling</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > On Wed, 2005-04-13 at 01:28, Eitan Zahavi wrote:</FONT>
<BR><FONT SIZE=2>> > > [EZ] This is true. Currently there is only one cause for the</FONT>
<BR><FONT SIZE=2>> > > un-healthy bits to be set - which are exactly as you point - these</FONT>
<BR><FONT SIZE=2>> > > traps. The point I was trying to make was that this bit is the</FONT>
<BR><FONT SIZE=2>> > > mechanism for flagging a port status is bad.</FONT>
<BR><FONT SIZE=2>> > ></FONT>
<BR><FONT SIZE=2>> > > What I did recommend was to write a "statistical" analysis of</FONT>
<BR><FONT SIZE=2>> Directed</FONT>
<BR><FONT SIZE=2>> > > Route packet drop - such that we can find the ports with a high drop</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> > > rate and mark them as un-healthy. If you mark every port that does</FONT>
<BR><FONT SIZE=2>> not</FONT>
<BR><FONT SIZE=2>> > > respond to a MAD as un-healthy you can suffer from flaky links</FONT>
<BR><FONT SIZE=2>> > > somewhere on the route to that port. Only analysis of the number of</FONT>
<BR><FONT SIZE=2>> > > good packets vs. dropped packets can lead you to the right bad port.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > The original proposal on this said the following:</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > "The OpenSM will implement a configurable policy (some number of</FONT>
<BR><FONT SIZE=2>> > consecutive lack of responses to SM requests). At the point of</FONT>
<BR><FONT SIZE=2>> > exhaustion of the timeout/retry strategy, that port will be marked as</FONT>
<BR><FONT SIZE=2>> > "bad" by OpenSM."</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > Any idea on what might make a good default threshold (for consecutive</FONT>
<BR><FONT SIZE=2>> > retries) ? Do you think there is no sufficient default ?</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > If a link is flaky and MADs can't get through, should it be used for</FONT>
<BR><FONT SIZE=2>> non</FONT>
<BR><FONT SIZE=2>> > MAD traffic ?</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > Also note that the proposal also said:</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > "Also, there could also be a periodic "ping" at a slower rate to check</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> > if the "bad" ports revive."</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > In terms of analysis of good v. errored and dropped packets (along the</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> > path to that node), there are OpenIB diagnostic tools to help with</FONT>
<BR><FONT SIZE=2>> this.</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > -- Hal</FONT>
</P>
</BODY>
</HTML>