<HTML dir=ltr><HEAD><TITLE>RE: [ofa-general] GPFS node loses IB-connection</TITLE>
<META http-equiv=Content-Type content="text/html; charset=unicode">
<META content="MSHTML 6.00.2900.3086" name=GENERATOR></HEAD>
<BODY>
<DIV id=idOWAReplyText61755 dir=ltr>
<DIV dir=ltr><FONT face=Arial color=#000000>After changing the switch timeout value, we never got the error again. Yesterday, we started a 24h stresstest. This test was succesfull. I think we can conclude that the problem is fixed now.</FONT></DIV>
<DIV dir=ltr><FONT face=Arial></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial>But, there is a strange message in de logs of the switch:</FONT></DIV>
<DIV dir=ltr>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM DELETE_MC_GROUP trap for GID=xx</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM CREATE_MC_GROUP trap for GID=xx</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM DELETE_MC_GROUP trap for GID=xx</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM CREATE_MC_GROUP trap for GID=xx</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[618]: %IB-6-INFO: Configuration caused by multicast membership change</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[618]: %IB-6-INFO: Configuration caused by multicast membership change</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt"></SPAN></FONT> </P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM DELETE_MC_GROUP trap for GID=yy</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM CREATE_MC_GROUP trap for GID=yy</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM DELETE_MC_GROUP trap for GID=yy</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[632]: %IB-6-INFO: Generate SM CREATE_MC_GROUP trap for GID=yy</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[618]: %IB-6-INFO: Configuration caused by multicast membership change</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt">Topspin-120sc ib_sm.x[618]: %IB-6-INFO: Configuration caused by multicast membership change</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN lang=EN-GB style="FONT-SIZE: 10pt"></SPAN></FONT> </P>
<P class=MsoNormal><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">With xx,yy = (e.g) ff:12:60:1b:ff:ff:</SPAN><SPAN lang=EN-GB>00:00:00:00:00:01</SPAN><SPAN lang=EN-GB>:ff:05:87:d9 but changing to different GIDs in the next group of loggings each belonging to the IB ports of the server HCA’s.</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial><SPAN lang=EN-GB></SPAN></FONT><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">This logging occurs every few minutes (not at a regular interval). </SPAN></FONT><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">Is there somewhere a Cisco manual available that describes or explains these messages? </SPAN></FONT><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">Or can anyone explain what is happening? And whether this can harm a setup that doesn't use multicast?</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">Greetz</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial><SPAN lang=EN-GB style="FONT-SIZE: 12pt">Koen</SPAN></FONT></P></DIV></DIV>
<DIV dir=ltr><BR>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>Van:</B> Scott Weitzenkamp (sweitzen) [mailto:sweitzen@cisco.com]<BR><B>Verzonden:</B> wo 23/05/2007 17:40<BR><B>Aan:</B> SEGERS Koen; Hal Rosenstock<BR><B>CC:</B> Clive Hall (clivhall); general-bounces@lists.openfabrics.org; general@lists.openfabrics.org<BR><B>Onderwerp:</B> RE: [ofa-general] GPFS node loses IB-connection<BR></FONT><BR></DIV>
<DIV>
<P><FONT size=2>Try 20 seconds, I'm curious if if you are barely crossing the 10-second<BR>threshold.<BR><BR>Scott<BR><BR>> -----Original Message-----<BR>> From: SEGERS Koen [<A href="mailto:Koen.SEGERS@VRT.BE">mailto:Koen.SEGERS@VRT.BE</A>]<BR>> Sent: Wednesday, May 23, 2007 8:39 AM<BR>> To: Scott Weitzenkamp (sweitzen); Hal Rosenstock<BR>> Cc: Clive Hall (clivhall);<BR>> general-bounces@lists.openfabrics.org; general@lists.openfabrics.org<BR>> Subject: RE: [ofa-general] GPFS node loses IB-connection<BR>><BR>> What value would you recommend then?<BR>><BR>> Koen<BR>><BR>> -----Oorspronkelijk bericht-----<BR>> Van: Scott Weitzenkamp (sweitzen) [<A href="mailto:sweitzen@cisco.com">mailto:sweitzen@cisco.com</A>]<BR>> Verzonden: woensdag 23 mei 2007 17:38<BR>> Aan: SEGERS Koen; Hal Rosenstock<BR>> CC: Clive Hall (clivhall); general-bounces@lists.openfabrics.org;<BR>> general@lists.openfabrics.org<BR>> Onderwerp: RE: [ofa-general] GPFS node loses IB-connection<BR>><BR>> The boot time of the host doesn't matter for this timeout. While the<BR>> host is booting, the IB link is down anyway.<BR>><BR>> Scott<BR>><BR>> > -----Original Message-----<BR>> > From: SEGERS Koen [<A href="mailto:Koen.SEGERS@VRT.BE">mailto:Koen.SEGERS@VRT.BE</A>]<BR>> > Sent: Wednesday, May 23, 2007 8:20 AM<BR>> > To: Hal Rosenstock; Scott Weitzenkamp (sweitzen)<BR>> > Cc: Clive Hall (clivhall);<BR>> > general-bounces@lists.openfabrics.org; general@lists.openfabrics.org<BR>> > Subject: RE: [ofa-general] GPFS node loses IB-connection<BR>> ><BR>> > After a whole day of stresstesting with the MAD renicing<BR>> turned on, we<BR>> > got the error once. So I think I should raise the timeout on<BR>> > the switch<BR>> > also.<BR>> ><BR>> > It takes about 2 minutes to boot the system. Do you agree<BR>> > that this is a<BR>> > good value for the timeout?<BR>> ><BR>> > Scott,<BR>> > Can you explain me the problem of the memlock?<BR>> ><BR>> > I saw that the SLES10 bug is only an issue in MVAPICH.<BR>> Since we didn't<BR>> > install this, the bug is not related to us. This is<BR>> correct, isn't it?<BR>> ><BR>> > Greetz<BR>> ><BR>> > Koen<BR>> ><BR>> > -----Oorspronkelijk bericht-----<BR>> > Van: Hal Rosenstock [<A href="mailto:halr@voltaire.com">mailto:halr@voltaire.com</A>]<BR>> > Verzonden: woensdag 23 mei 2007 16:12<BR>> > Aan: Scott "Weitzenkamp (sweitzen)<BR>> > CC: SEGERS Koen; Clive Hall (clivhall);<BR>> > general-bounces@lists.openfabrics.org; general@lists.openfabrics.org<BR>> > Onderwerp: RE: [ofa-general] GPFS node loses IB-connection<BR>> ><BR>> > On Wed, 2007-05-23 at 09:51, Scott Weitzenkamp (sweitzen) wrote:<BR>> > > No C code changes, just a few config file changes<BR>> (RENICE_IB_MAD=yes<BR>> > in<BR>> > > openib.conf,<BR>> ><BR>> > Does the host really not respond to MAD requests for over 10<BR>> > seconds in<BR>> > some cases ?<BR>> ><BR>> > -- Hal<BR>> ><BR>> > > memlock in /etc/security/limits.conf, fix /etc/hosts on<BR>> > > SLES10 for bug 267, etc.).<BR>> > ><BR>> > > Scott Weitzenkamp<BR>> > > SQA and Release Manager<BR>> > > Server Virtualization Business Unit<BR>> > > Cisco Systems<BR>> > > <BR>> > ><BR>> > > > -----Original Message-----<BR>> > > > From: SEGERS Koen [<A href="mailto:Koen.SEGERS@VRT.BE">mailto:Koen.SEGERS@VRT.BE</A>]<BR>> > > > Sent: Wednesday, May 23, 2007 6:48 AM<BR>> > > > To: Scott Weitzenkamp (sweitzen); Clive Hall (clivhall)<BR>> > > > Cc: Shirley Ma; Ami Perlmutter;<BR>> > > > general@lists.openfabrics.org;<BR>> > general-bounces@lists.openfabrics.org<BR>> > > > Subject: RE: [ofa-general] GPFS node loses IB-connection<BR>> > > ><BR>> > > > This far, all tests seem to work.<BR>> > > ><BR>> > > > Thanks for the help!<BR>> > > ><BR>> > > > Scott,<BR>> > > > Are there more bugfixes that cisco does in its rpms?<BR>> > > ><BR>> > > > Greetz<BR>> > > ><BR>> > > > Koen<BR>> > > ><BR>> > > > -----Oorspronkelijk bericht-----<BR>> > > > Van: Scott Weitzenkamp (sweitzen) [<A href="mailto:sweitzen@cisco.com">mailto:sweitzen@cisco.com</A>]<BR>> > > > Verzonden: woensdag 23 mei 2007 0:39<BR>> > > > Aan: SEGERS Koen; Scott Weitzenkamp (sweitzen); Clive Hall<BR>> > (clivhall)<BR>> > > > CC: Shirley Ma; Ami Perlmutter; general@lists.openfabrics.org;<BR>> > > > general-bounces@lists.openfabrics.org<BR>> > > > Onderwerp: RE: [ofa-general] GPFS node loses IB-connection<BR>> > > ><BR>> > > > It's not so much pinging every 10 seconds as expecting a<BR>> > > > response within<BR>> > > > 10 seconds (Clive, correct me if I'm wrong).<BR>> > > ><BR>> > > > You only need to do 1) or 2), not both. Cisco configures 1)<BR>> > > > in the OFED<BR>> > > > binary RPMs we release at<BR>> > > > <A href="http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux">http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux</A>. I<BR>> > > > prefer to have<BR>> > > > the host be more responsive.<BR>> > > ><BR>> > > ><BR>> > > > Scott Weitzenkamp<BR>> > > > SQA and Release Manager<BR>> > > > Server Virtualization Business Unit<BR>> > > > Cisco Systems<BR>> > > > <BR>> > > ><BR>> > > > > -----Original Message-----<BR>> > > > > From: Koen Segers [<A href="mailto:koen.segers@VRT.BE">mailto:koen.segers@VRT.BE</A>]<BR>> > > > > Sent: Tuesday, May 22, 2007 3:35 PM<BR>> > > > > To: Scott Weitzenkamp (sweitzen)<BR>> > > > > Cc: Shirley Ma; Ami Perlmutter;<BR>> > > > > general@lists.openfabrics.org;<BR>> > general-bounces@lists.openfabrics.org<BR>> > > > > Subject: RE: [ofa-general] GPFS node loses IB-connection<BR>> > > > ><BR>> > > > > If I understand it wright, the switch is actually polling<BR>> > > > > (=pinging) the<BR>> > > > > interfaces every 10s. This means that when the interface is<BR>> > handling<BR>> > > > > other traffic, the poll can fail and the port could be<BR>> > > > > considered out of<BR>> > > > > service. My question is then: "How can the timeout be reached<BR>> > while<BR>> > > > > packets are being sent/received to/from the interface?"<BR>> > > > ><BR>> > > > > Anyway, what timeout-value would you recommend for<BR>> us? And why?<BR>> > > > ><BR>> > > > > To recapitulate: these are the actions I'll take tomorrow<BR>> > > > > 1) change the MAD niceness of the servers<BR>> > > > > 2) change the timeout on the switches<BR>> > > > ><BR>> > > > > Are these changes sufficient for the HCA's to keep<BR>> > their ports in<BR>> > > > > PORT_ACTIVE state?<BR>> > > > ><BR>> > > > > Regards,<BR>> > > > ><BR>> > > > > Koen<BR>> > > > ><BR>> > > > > On Tue, 2007-05-22 at 12:59 -0700, Scott Weitzenkamp<BR>> > > > (sweitzen) wrote:<BR>> > > > > > Yes, you can tune it. Here's an example via the switch CLI:<BR>> > > > > > <BR>> > > > > > SFS-7000D(config)# ib sm subnet-prefix<BR>> fe:80:00:00:00:00:00:00<BR>> > > > > > node-timeout <value><BR>> > > > > ><BR>> > > > > > The default is 10 seconds, it can be configured up to<BR>> > > > 2000 seconds.<BR>> > > > > > If a HCA is completely unresponsive for longer than the<BR>> > > > node-timeout<BR>> > > > > > value, then we consider that HCA out of service.<BR>> > > > > > <BR>> > > > > > Scott Weitzenkamp<BR>> > > > > > SQA and Release Manager<BR>> > > > > > Server Virtualization Business Unit<BR>> > > > > > Cisco Systems<BR>> > > > > > <BR>> > > > > ><BR>> > > > > > <BR>> > > > > > <BR>> > > > > ______________________________________________________________<BR>> > > > > > From: Shirley Ma [<A href="mailto:xma@us.ibm.com">mailto:xma@us.ibm.com</A>]<BR>> > > > > > Sent: Tuesday, May 22, 2007 11:30 AM<BR>> > > > > > To: koen.segers@VRT.BE<BR>> > > > > > Cc: Ami Perlmutter; general@lists.openfabrics.org;<BR>> > > > > > general-bounces@lists.openfabrics.org; Scott<BR>> > Weitzenkamp<BR>> > > > > > (sweitzen)<BR>> > > > > > Subject: RE: [ofa-general] GPFS node loses<BR>> > IB-connection<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > Koen,<BR>> > > > > > <BR>> > > > > > So it is most likely you hit the same bug as<BR>> > 229 (Scott<BR>> > > > > > pointed out earlier). The same workaround might<BR>> > > > work for you<BR>> > > > > > by renicing ib_mad as Scott suggested.<BR>> > > > > > <BR>> > > > > > I think this should be a SM query timeout<BR>> > tunable value<BR>> > in<BR>> > > > > > Cisco SM. Am I right, Scott?<BR>> > > > > > <BR>> > > > > > Thanks<BR>> > > > > > Shirley Ma<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > Inactive hide details for Koen Segers<BR>> > > > > <koen.segers@VRT.BE>Koen<BR>> > > > > > Segers <koen.segers@VRT.BE><BR>> > > > > > <BR>> > > > > > <BR>> > > > > > Koen Segers<BR>> > > > > <koen.segers@VRT.BE><BR>> > > > > > <BR>> > > > > > 05/22/07 11:14 AM<BR>> > > > > > Please respond to<BR>> > > > > > koen.segers@VRT.BE<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > To<BR>> > > > > > <BR>> > > > > > Shirley<BR>> > > > > > Ma/Beaverton/IBM@IBMUS<BR>> > > > > > <BR>> > > > > > cc<BR>> > > > > > <BR>> > > > > > Ami Perlmutter<BR>> > > > > > <amip@dev.mellanox.co.il>,<BR>> > > > > general@lists.openfabrics.org,<BR>> > general-bounces@lists.openfabrics.org<BR>> > > > > > <BR>> > > > > > Subject<BR>> > > > > > <BR>> > > > > > RE:<BR>> > > > > > [ofa-general]<BR>> > > > > > GPFS node loses<BR>> > > > > > IB-connection<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > Hi,<BR>> > > > > > <BR>> > > > > > It is the Cisco SM.<BR>> > > > > > <BR>> > > > > > SFS-7000P> show version<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > ==============================================================<BR>> > > > > ==================<BR>> > > > > > System Version Information<BR>> > > > > > <BR>> > > > > ==============================================================<BR>> > > > > ==================<BR>> > > > > > system-version : SFS-7000P TopspinOS<BR>> > > > 2.9.0 releng<BR>> > > > > > #147<BR>> > > > > > 10/25/2006 02:01:32<BR>> > > > > > contact : tac@cisco.com<BR>> > > > > > name : SFS-7000P<BR>> > > > > > location : 170 West Tasman Drive,<BR>> > > > > San Jose, CA<BR>> > > > > > 95134<BR>> > > > > > up-time : 11(d):7(h):49(m):3(s)<BR>> > > > > > last-change : none<BR>> > > > > > last-config-save : none<BR>> > > > > > action : none<BR>> > > > > > result : none<BR>> > > > > > oper-mode : normal<BR>> > > > > > <BR>> > > > > > There is also a command that gives the SM version,<BR>> > > > > but I can't<BR>> > > > > > find it<BR>> > > > > > right now.<BR>> > > > > > <BR>> > > > > > On Tue, 2007-05-22 at 09:45 -0700, Shirley Ma wrote:<BR>> > > > > > > Hello Koen,<BR>> > > > > > ><BR>> > > > > > > From the switch log, it looks a SM issue to me.<BR>> > > > > The node was<BR>> > > > > > kicked<BR>> > > > > > > out of the membership. Which SM you are<BR>> > using in your<BR>> > > > > > fabric?<BR>> > > > > > ><BR>> > > > > > > Thanks<BR>> > > > > > > Shirley Ma<BR>> > > > > > ><BR>> > > > > > *** Disclaimer ***<BR>> > > > > > <BR>> > > > > > Vlaamse Radio- en Televisieomroep<BR>> > > > > > Auguste Reyerslaan 52, 1043 Brussel<BR>> > > > > > <BR>> > > > > > nv van publiek recht<BR>> > > > > > BTW BE 0244.142.664<BR>> > > > > > RPR Brussel<BR>> > > > > > <A href="http://www.vrt.be/disclaimer">http://www.vrt.be/disclaimer</A><BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > *** Disclaimer ***<BR>> > > > ><BR>> > > > > Vlaamse Radio- en Televisieomroep<BR>> > > > > Auguste Reyerslaan 52, 1043 Brussel<BR>> > > > ><BR>> > > > > nv van publiek recht<BR>> > > > > BTW BE 0244.142.664<BR>> > > > > RPR Brussel<BR>> > > > > <A href="http://www.vrt.be/disclaimer">http://www.vrt.be/disclaimer</A><BR>> > > > > <BR>> > > > ><BR>> > > > *** Disclaimer ***<BR>> > > ><BR>> > > > Vlaamse Radio- en Televisieomroep<BR>> > > > Auguste Reyerslaan 52, 1043 Brussel<BR>> > > ><BR>> > > > nv van publiek recht<BR>> > > > BTW BE 0244.142.664<BR>> > > > RPR Brussel<BR>> > > > <A href="http://www.vrt.be/disclaimer">http://www.vrt.be/disclaimer</A><BR>> > > > <BR>> > > ><BR>> > > _______________________________________________<BR>> > > general mailing list<BR>> > > general@lists.openfabrics.org<BR>> > > <A href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</A><BR>> > ><BR>> > > To unsubscribe, please visit<BR>> > <A href="http://openib.org/mailman/listinfo/openib-general">http://openib.org/mailman/listinfo/openib-general</A><BR>> ><BR>> > *** Disclaimer ***<BR>> ><BR>> > Vlaamse Radio- en Televisieomroep<BR>> > Auguste Reyerslaan 52, 1043 Brussel<BR>> ><BR>> > nv van publiek recht<BR>> > BTW BE 0244.142.664<BR>> > RPR Brussel<BR>> > <A href="http://www.vrt.be/disclaimer">http://www.vrt.be/disclaimer</A><BR>> > <BR>> ><BR>> *** Disclaimer ***<BR>><BR>> Vlaamse Radio- en Televisieomroep<BR>> Auguste Reyerslaan 52, 1043 Brussel<BR>><BR>> nv van publiek recht<BR>> BTW BE 0244.142.664<BR>> RPR Brussel<BR>> <A href="http://www.vrt.be/disclaimer">http://www.vrt.be/disclaimer</A><BR>> <BR>><BR></FONT></P></DIV>*** Disclaimer ***<br><br>Vlaamse Radio- en Televisieomroep<br>Auguste Reyerslaan 52, 1043 Brussel<br><br>nv van publiek recht<br>BTW BE 0244.142.664<br>RPR Brussel<br>http://www.vrt.be/disclaimer<br> <br></BODY></HTML>