Hi Rafael,<br><br>
<div><span class="gmail_quote">On 8/25/09, <b class="gmail_sendername">Rafael David Tinoco</b> <<a href="mailto:Rafael.Tinoco@sun.com">Rafael.Tinoco@sun.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff">Hello Hal, <br><br>Bellow...
<div><span class="e" id="q_123548d28df86c71_1"><br><br>Hal Rosenstock wrote:
<blockquote type="cite"><br><br>
<div><span class="gmail_quote">On 8/24/09, <b class="gmail_sendername">Rafael David Tinoco</b> <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Rafael.Tinoco@sun.com" target="_blank">Rafael.Tinoco@sun.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<div text="#000000" bgcolor="#ffffff"><font size="-1"><font face="Consolas">Hello, <br><br>I'm installing an HPC cluster using 2 Sun Blades 6048 with QNEMs (2 asics each, 8 qnems).<br>They are configured in a MESH topology.<br>
I'm using Centos 5.3, OFED 1.4.1 and kernel 2.6.18-128.el5.<br><br>I'm booting PXE from IB, my initrd image is bringing the ib0 interface, getting the squashfs image and mounting with aufs.<br><br>The problem is.. When booting more then 60 nodes, I start to get above errors on subnet manager.<br>
And the problem seems to be intermitent, because each time it gives errors on different path.<br><br>Any ideas ?<br><br>Aug 24 15:36:19 713836 [48D7D940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713838 [48D7D940] 0x02 -> __osm_state_mgr_report_new_ports: Discovered new port with GUID:0x50800200008d9381 LID range [78,78] of node:b03n06 HCA-1<br>Aug 24 15:36:19 713840 [48D7D940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713842 [48D7D940] 0x02 -> __osm_state_mgr_report_new_ports: Discovered new port with GUID:0x50800200008d4689 LID range [76,76] of node:b03n04 HCA-1<br>Aug 24 15:36:19 713845 [48D7D940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713847 [48D7D940] 0x02 -> __osm_state_mgr_report_new_ports: Discovered new port with GUID:0x50800200008e5191 LID range [82,82] of node:b03n11 HCA-1<br>Aug 24 15:36:19 713849 [48D7D940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713866 [48D7D940] 0x02 -> __osm_state_mgr_report_new_ports: Discovered new port with GUID:0x50800200008d94c9 LID range [80,80] of node:b03n08 HCA-1<br>Aug 24 15:36:19 713869 [48D7D940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713871 [48D7D940] 0x02 -> __osm_state_mgr_report_new_ports: Discovered new port with GUID:0x50800200008daedd LID range [83,83] of node:b03n12 HCA-1<br>Aug 24 15:36:19 714782 [48D7D940] 0x02 -> SUBNET UP<br>
Aug 24 15:36:19 714805 [48D7D940] 0x01 -> __osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A) port 19. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714812 [48D7D940] 0x01 -> Directed Path Dump of 4 hop path:<br> Path = 0,1,15,15,15<br>Aug 24 15:36:19 714822 [48D7D940] 0x01 -> __osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A) port 21. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714827 [48D7D940] 0x01 -> Directed Path Dump of 4 hop path:<br> Path = 0,1,15,15,15<br>Aug 24 15:36:19 714831 [48D7D940] 0x01 -> __osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A) port 25. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714835 [48D7D940] 0x01 -> Directed Path Dump of 4 hop path:<br> Path = 0,1,15,15,15<br>Aug 24 15:36:20 514302 [4977E940] 0x01 -> umad_receiver: ERR 5409: send completed with error (method=0x1 attr=0x15 trans_id=0x4700036595) -- dropping<br>
Aug 24 15:36:20 514321 [4977E940] 0x01 -> umad_receiver: ERR 5411: DR SMP Hop Ptr: 0x0<br>Aug 24 15:36:20 514328 [4977E940] 0x01 -> Received SMP on a 5 hop path:<br> Initial path = 0,0,0,0,0,0<br> Return path = 0,0,0,0,0,0<br>
Aug 24 15:36:20 514333 [4977E940] 0x01 -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT)<br>Aug 24 15:36:20 514352 [4977E940] 0x01 -> SMP dump:<br> base_ver................0x1<br>
mgmt_class..............0x81<br> class_ver...............0x1<br> method..................0x1 (SubnGet)<br> D bit...................0x0<br> status..................0x0<br>
hop_ptr.................0x0<br> hop_count...............0x5<br> trans_id................0x36595<br> attr_id.................0x15 (PortInfo)<br> resv....................0x0<br>
attr_mod................0x0<br> m_key...................0x0000000000000000<br> dr_slid.................65535<br> dr_dlid.................65535<br><br> Initial path: 0,1,15,15,15,19<br>
Return path: 0,0,0,0,0,0<br> Reserved: [0][0][0][0][0][0][0]<br><br> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br><br> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
<br> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br><br> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br><br>Aug 24 15:36:20 514364 [4977E940] 0x01 -> umad_receiver: ERR 5409: send completed with error (method=0x1 attr=0x15 trans_id=0x4700036596) -- dropping<br>
Aug 24 15:36:20 514367 [4977E940] 0x01 -> umad_receiver: ERR 5411: DR SMP Hop Ptr: 0x0<br>Aug 24 15:36:20 514372 [4977E940] 0x01 -> Received SMP on a 5 hop path:<br> Initial path = 0,0,0,0,0,0<br> Return path = 0,0,0,0,0,0<br>
Aug 24 15:36:20 514375 [4977E940] 0x01 -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT)<br>Aug 24 15:36:20 514391 [4977E940] 0x01 -> SMP dump:<br> base_ver................0x1<br>
mgmt_class..............0x81<br> class_ver...............0x1<br> method..................0x1 (SubnGet)<br> D bit...................0x0<br> status..................0x0<br>
hop_ptr.................0x0<br> hop_count...............0x5<br> trans_id................0x36596<br> attr_id.................0x15 (PortInfo)<br> resv....................0x0<br>
....</font></font></div></blockquote>
<div> </div>
<div>These errors are transient as you indicate. They mean that some node has brought the link physically up but there is no SMA at the remote side of the link. The different paths are paths to the HCAs. This occurs during PXE boot as the node transitions from the boot ROM to the Linux environment.</div>
<div> </div></div></blockquote></span></div>They are transient.. but sometimes opensm hangs with the same message and loops this errors messages. </div></blockquote>
<div> </div>
<div>Are you sure OpenSM hangs ? If so, any idea where ?</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff">First I was using centos 5.3 kernel with updates .. and the IPoIB stopped working after these messages.</div></blockquote>
<div> </div>
<div>Any specifics ?</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff">Using the "vanilla" centos 5.3 kernel solved this issue.<br>But SOMETIMES, booting the nodes, these messages appear and dont go away.</div></blockquote>
<div> </div>
<div>In those cases, do the nodes succesfully boot up ?</div>
<div> </div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff"><span class="q">
<blockquote type="cite">
<div>
<div>Other than these messages, do things seem to work in terms of the end nodes ?</div></div></blockquote></span>They seem to work with vanilla kernel. Even with the messages, no problems reaching the nodes so far.</div>
</blockquote>
<div> </div>
<div>Do your ULPs work (like IPoIB, etc.) ?</div>
<div> </div>
<div>-- Hal</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff">Tks<br><span class="sg"><br>Rafael Tinoco</span><span class="q"><br>
<blockquote type="cite">
<div>
<div> </div>
<div>-- Hal</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">_______________________________________________<br>general mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:general@lists.openfabrics.org" target="_blank">general@lists.openfabrics.org</a><br>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br><br>To unsubscribe, please visit <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://openib.org/mailman/listinfo/openib-general" target="_blank">http://openib.org/mailman/listinfo/openib-general</a><br>
</blockquote></div><br></blockquote></span></div></blockquote></div><br>