<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hello Hal, <br>
<br>
Bellow...<br>
<br>
Hal Rosenstock wrote:
<blockquote
cite="mid:f0e08f230908251504m4aec4233ke6aa5b009ce1232c@mail.gmail.com"
type="cite"><br>
<br>
<div><span class="gmail_quote">On 8/24/09, <b
class="gmail_sendername">Rafael David Tinoco</b> <<a
moz-do-not-send="true"
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:Rafael.Tinoco@sun.com" target="_blank">Rafael.Tinoco@sun.com</a>>
wrote:</span>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0px 0px 0px 0.8ex; padding-left: 1ex;">
<div text="#000000" bgcolor="#ffffff"><font size="-1"><font
face="Consolas">Hello, <br>
<br>
I'm installing an HPC cluster using 2 Sun Blades 6048 with QNEMs (2
asics each, 8 qnems).<br>
They are configured in a MESH topology.<br>
I'm using Centos 5.3, OFED 1.4.1 and kernel 2.6.18-128.el5.<br>
<br>
I'm booting PXE from IB, my initrd image is bringing the ib0 interface,
getting the squashfs image and mounting with aufs.<br>
<br>
The problem is.. When booting more then 60 nodes, I start to get above
errors on subnet manager.<br>
And the problem seems to be intermitent, because each time it gives
errors on different path.<br>
<br>
Any ideas ?<br>
<br>
Aug 24 15:36:19 713836 [48D7D940] 0x02 -> osm_report_notice:
Reporting Generic Notice type:3 num:64 (GID in service) from LID:1
GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713838 [48D7D940] 0x02 ->
__osm_state_mgr_report_new_ports: Discovered new port with
GUID:0x50800200008d9381 LID range [78,78] of node:b03n06 HCA-1<br>
Aug 24 15:36:19 713840 [48D7D940] 0x02 -> osm_report_notice:
Reporting Generic Notice type:3 num:64 (GID in service) from LID:1
GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713842 [48D7D940] 0x02 ->
__osm_state_mgr_report_new_ports: Discovered new port with
GUID:0x50800200008d4689 LID range [76,76] of node:b03n04 HCA-1<br>
Aug 24 15:36:19 713845 [48D7D940] 0x02 -> osm_report_notice:
Reporting Generic Notice type:3 num:64 (GID in service) from LID:1
GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713847 [48D7D940] 0x02 ->
__osm_state_mgr_report_new_ports: Discovered new port with
GUID:0x50800200008e5191 LID range [82,82] of node:b03n11 HCA-1<br>
Aug 24 15:36:19 713849 [48D7D940] 0x02 -> osm_report_notice:
Reporting Generic Notice type:3 num:64 (GID in service) from LID:1
GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713866 [48D7D940] 0x02 ->
__osm_state_mgr_report_new_ports: Discovered new port with
GUID:0x50800200008d94c9 LID range [80,80] of node:b03n08 HCA-1<br>
Aug 24 15:36:19 713869 [48D7D940] 0x02 -> osm_report_notice:
Reporting Generic Notice type:3 num:64 (GID in service) from LID:1
GID:fe80::5080:200:8d:9931<br>
Aug 24 15:36:19 713871 [48D7D940] 0x02 ->
__osm_state_mgr_report_new_ports: Discovered new port with
GUID:0x50800200008daedd LID range [83,83] of node:b03n12 HCA-1<br>
Aug 24 15:36:19 714782 [48D7D940] 0x02 -> SUBNET UP<br>
Aug 24 15:36:19 714805 [48D7D940] 0x01 ->
__osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for
node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A)
port 19. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714812 [48D7D940] 0x01 -> Directed Path Dump of 4
hop path:<br>
Path = 0,1,15,15,15<br>
Aug 24 15:36:19 714822 [48D7D940] 0x01 ->
__osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for
node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A)
port 21. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714827 [48D7D940] 0x01 -> Directed Path Dump of 4
hop path:<br>
Path = 0,1,15,15,15<br>
Aug 24 15:36:19 714831 [48D7D940] 0x01 ->
__osm_state_mgr_light_sweep_start: ERR 3315: Unknown remote side for
node 0x0021283a85260040(Sun Blade 6048 InfiniBand QDR Switched NEM I4A)
port 25. Adding to light sweep sampling list<br>
Aug 24 15:36:19 714835 [48D7D940] 0x01 -> Directed Path Dump of 4
hop path:<br>
Path = 0,1,15,15,15<br>
Aug 24 15:36:20 514302 [4977E940] 0x01 -> umad_receiver: ERR 5409:
send completed with error (method=0x1 attr=0x15 trans_id=0x4700036595)
-- dropping<br>
Aug 24 15:36:20 514321 [4977E940] 0x01 -> umad_receiver: ERR 5411:
DR SMP Hop Ptr: 0x0<br>
Aug 24 15:36:20 514328 [4977E940] 0x01 -> Received SMP on a 5 hop
path:<br>
Initial path = 0,0,0,0,0,0<br>
Return path = 0,0,0,0,0,0<br>
Aug 24 15:36:20 514333 [4977E940] 0x01 ->
__osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error
(IB_TIMEOUT)<br>
Aug 24 15:36:20 514352 [4977E940] 0x01 -> SMP dump:<br>
base_ver................0x1<br>
mgmt_class..............0x81<br>
class_ver...............0x1<br>
method..................0x1 (SubnGet)<br>
D bit...................0x0<br>
status..................0x0<br>
hop_ptr.................0x0<br>
hop_count...............0x5<br>
trans_id................0x36595<br>
attr_id.................0x15 (PortInfo)<br>
resv....................0x0<br>
attr_mod................0x0<br>
m_key...................0x0000000000000000<br>
dr_slid.................65535<br>
dr_dlid.................65535<br>
<br>
Initial path: 0,1,15,15,15,19<br>
Return path: 0,0,0,0,0,0<br>
Reserved: [0][0][0][0][0][0][0]<br>
<br>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
<br>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
<br>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
<br>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
<br>
Aug 24 15:36:20 514364 [4977E940] 0x01 -> umad_receiver: ERR 5409:
send completed with error (method=0x1 attr=0x15 trans_id=0x4700036596)
-- dropping<br>
Aug 24 15:36:20 514367 [4977E940] 0x01 -> umad_receiver: ERR 5411:
DR SMP Hop Ptr: 0x0<br>
Aug 24 15:36:20 514372 [4977E940] 0x01 -> Received SMP on a 5 hop
path:<br>
Initial path = 0,0,0,0,0,0<br>
Return path = 0,0,0,0,0,0<br>
Aug 24 15:36:20 514375 [4977E940] 0x01 ->
__osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error
(IB_TIMEOUT)<br>
Aug 24 15:36:20 514391 [4977E940] 0x01 -> SMP dump:<br>
base_ver................0x1<br>
mgmt_class..............0x81<br>
class_ver...............0x1<br>
method..................0x1 (SubnGet)<br>
D bit...................0x0<br>
status..................0x0<br>
hop_ptr.................0x0<br>
hop_count...............0x5<br>
trans_id................0x36596<br>
attr_id.................0x15 (PortInfo)<br>
resv....................0x0<br>
....</font></font></div>
</blockquote>
<div> </div>
<div>These errors are transient as you indicate. They mean that some
node has brought the link physically up but there is no SMA at the
remote side of the link. The different paths are paths to the HCAs.
This occurs during PXE boot as the node transitions from the boot ROM
to the Linux environment.</div>
<div> </div>
</div>
</blockquote>
They are transient.. but sometimes opensm hangs with the same message
and loops this errors messages. <br>
First I was using centos 5.3 kernel with updates .. and the IPoIB
stopped working after these messages.<br>
Using the "vanilla" centos 5.3 kernel solved this issue.<br>
But SOMETIMES, booting the nodes, these messages appear and dont go
away.<br>
<blockquote
cite="mid:f0e08f230908251504m4aec4233ke6aa5b009ce1232c@mail.gmail.com"
type="cite">
<div>
<div>Other than these messages, do things seem to work in terms of
the end nodes ?</div>
</div>
</blockquote>
They seem to work with vanilla kernel. Even with the messages, no
problems reaching the nodes so far.<br>
<br>
Tks<br>
<br>
Rafael Tinoco<br>
<blockquote
cite="mid:f0e08f230908251504m4aec4233ke6aa5b009ce1232c@mail.gmail.com"
type="cite">
<div>
<div> </div>
<div>-- Hal</div>
<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0px 0px 0px 0.8ex; padding-left: 1ex;">_______________________________________________<br>
general mailing list<br>
<a moz-do-not-send="true"
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:general@lists.openfabrics.org" target="_blank">general@lists.openfabrics.org</a><br>
<a moz-do-not-send="true"
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general"
target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
<br>
To unsubscribe, please visit <a moz-do-not-send="true"
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://openib.org/mailman/listinfo/openib-general"
target="_blank">http://openib.org/mailman/listinfo/openib-general</a><br>
</blockquote>
</div>
<br>
</blockquote>
</body>
</html>