<br><font size=2 face="sans-serif">Hal,</font>
<br>
<br><font size=2 face="sans-serif">I rebuilt the opensm executable with
the patch you provided. The patch fixes (or avoids) the segmentation
fault and opensm comes up and runs. However, the link is still not
becoming operational. On the local side it goes to ARMED, and
on the remote side it goes to INIT. The osm.log seems to show that
the MAD packets are timing out. Here is the first part of the file,
it just repeats after this at one minute intervals.</font>
<br>
<br><font size=2><tt>[koa] (ib) root> cat /var/log/osm.log</tt></font>
<br><font size=2><tt>May 26 14:05:43 369104 [8EFC3D00] -> OpenSM Rev:openib-1.2.0
OpenIB svn Exported revision</tt></font>
<br><font size=2><tt>May 26 14:05:43 369260 [0000] -> OpenSM Rev:openib-1.2.0
OpenIB svn Exported revision</tt></font>
<br>
<br><font size=2><tt>May 26 14:05:43 370571 [8EFC3D00] -> osm_report_notice:
Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe8000000000000</tt></font>
<br><font size=2><tt>0,0x0000000000000000</tt></font>
<br><font size=2><tt>May 26 14:05:43 370631 [8EFC3D00] -> osm_report_notice:
Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe8000000000000</tt></font>
<br><font size=2><tt>0,0x0000000000000000</tt></font>
<br><font size=2><tt>May 26 14:05:43 373005 [8EFC3D00] -> osm_vendor_bind:
Binding to port 0x2c90200216dc5</tt></font>
<br><font size=2><tt>May 26 14:05:43 374685 [8EFC3D00] -> osm_vendor_bind:
Binding to port 0x2c90200216dc5</tt></font>
<br><font size=2><tt>May 26 14:05:44 172028 [44007960] -> umad_receiver:
ERR 5409: send completed with error (method=0x1 attr=0x11 trans_id=0x1239)
-- dr</tt></font>
<br><font size=2><tt>opping</tt></font>
<br><font size=2><tt>May 26 14:05:44 172070 [44007960] -> umad_receiver:
ERR 5411: DR SMP</tt></font>
<br><font size=2><tt>May 26 14:05:44 172083 [44007960] -> __osm_sm_mad_ctrl_send_err_cb:
ERR 3113: MAD completed in error (IB_TIMEOUT)</tt></font>
<br><font size=2><tt>May 26 14:05:44 172148 [44007960] -> SMP dump:</tt></font>
<br><font size=2><tt>
base_ver................0x1</tt></font>
<br><font size=2><tt>
mgmt_class..............0x81</tt></font>
<br><font size=2><tt>
class_ver...............0x1</tt></font>
<br><font size=2><tt>
method..................0x1
(SubnGet)</tt></font>
<br><font size=2><tt>
D bit...................0x0</tt></font>
<br><font size=2><tt>
status..................0x0</tt></font>
<br><font size=2><tt>
hop_ptr.................0x0</tt></font>
<br><font size=2><tt>
hop_count...............0x1</tt></font>
<br><font size=2><tt>
trans_id................0x1239</tt></font>
<br><font size=2><tt>
attr_id.................0x11
(NodeInfo)</tt></font>
<br><font size=2><tt>
resv....................0x0</tt></font>
<br><font size=2><tt>
attr_mod................0x0</tt></font>
<br><font size=2><tt>
m_key...................0x0000000000000000</tt></font>
<br><font size=2><tt>
dr_slid.................0xFFFF</tt></font>
<br><font size=2><tt>
dr_dlid.................0xFFFF</tt></font>
<br>
<br><font size=2><tt>
Initial path: [0][1]</tt></font>
<br><font size=2><tt>
Return path: [0][0]</tt></font>
<br><font size=2><tt>
Reserved:
[0][0][0][0][0][0][0]</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>May 26 14:05:44 172199 [42003960] -> osm_drop_mgr_process:
ERR 0108: Unknown remote side for node 0x0002c90200216dc4 port 1. Adding</tt></font>
<br><font size=2><tt>to light sweep sampling list</tt></font>
<br><font size=2><tt>May 26 14:05:44 172240 [42003960] -> Directed Path
Dump of 0 hop path:</tt></font>
<br><font size=2><tt>
Path = [0]</tt></font>
<br><font size=2><tt>May 26 14:05:44 172256 [0000] -> Entering MASTER
state</tt></font>
<br>
<br><font size=2><tt>May 26 14:05:44 179081 [0000] -> SUBNET UP</tt></font>
<br>
<br><font size=2><tt>May 26 14:05:54 180461 [44007960] -> umad_receiver:
ERR 5409: send completed with error (method=0x1 attr=0x11 trans_id=0x1240)
-- dr</tt></font>
<br><font size=2><tt>opping</tt></font>
<br><font size=2><tt>May 26 14:05:54 180515 [44007960] -> umad_receiver:
ERR 5411: DR SMP</tt></font>
<br><font size=2><tt>May 26 14:05:54 180528 [44007960] -> __osm_sm_mad_ctrl_send_err_cb:
ERR 3113: MAD completed in error (IB_TIMEOUT)</tt></font>
<br><font size=2><tt>May 26 14:05:54 180569 [44007960] -> SMP dump:</tt></font>
<br><font size=2><tt>
base_ver................0x1</tt></font>
<br><font size=2><tt>
mgmt_class..............0x81</tt></font>
<br><font size=2><tt>
class_ver...............0x1</tt></font>
<br><font size=2><tt>
method..................0x1
(SubnGet)</tt></font>
<br><font size=2><tt>
D bit...................0x0</tt></font>
<br><font size=2><tt>
status..................0x0</tt></font>
<br><font size=2><tt>
hop_ptr.................0x0</tt></font>
<br><font size=2><tt>
hop_count...............0x1</tt></font>
<br><font size=2><tt>
trans_id................0x1240</tt></font>
<br><font size=2><tt>
attr_id.................0x11
(NodeInfo)</tt></font>
<br><font size=2><tt>
resv....................0x0</tt></font>
<br><font size=2><tt>
attr_mod................0x0</tt></font>
<br><font size=2><tt>
m_key...................0x0000000000000000</tt></font>
<br><font size=2><tt>
dr_slid.................0xFFFF</tt></font>
<br><font size=2><tt>
dr_dlid.................0xFFFF</tt></font>
<br>
<br><font size=2><tt>
Initial path: [0][1]</tt></font>
<br><font size=2><tt>
Return path: [0][0]</tt></font>
<br><font size=2><tt>
Reserved:
[0][0][0][0][0][0][0]</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>
00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00</tt></font>
<br>
<br><font size=2><tt>May 26 14:05:54 180624 [42003960] -> osm_drop_mgr_process:
ERR 0108: Unknown remote side for node 0x0002c90200216dc4 port 1. Adding</tt></font>
<br><font size=2><tt>to light sweep sampling list</tt></font>
<br><font size=2><tt>May 26 14:05:54 180649 [42003960] -> Directed Path
Dump of 0 hop path:</tt></font>
<br><font size=2><tt>
Path = [0]</tt></font>
<br>
<br>
<br><font size=2 face="sans-serif">The physical link appears to be up:
here are the ibstat, ibstatus results for both sides:</font>
<br>
<br><font size=2 face="sans-serif">Local system</font>
<br>
<br><font size=2><tt>[koa] (ib) root> ibstat</tt></font>
<br><font size=2><tt>CA 'mthca0'</tt></font>
<br><font size=2><tt> CA type: MT25204</tt></font>
<br><font size=2><tt> Number of ports: 1</tt></font>
<br><font size=2><tt> Firmware version: 1.0.800</tt></font>
<br><font size=2><tt> Hardware version: a0</tt></font>
<br><font size=2><tt> Node GUID: 0x0002c90200216dc4</tt></font>
<br><font size=2><tt> System image GUID: 0x0002c90200216dc7</tt></font>
<br><font size=2><tt> Port 1:</tt></font>
<br><font size=2><tt>
State: Armed</tt></font>
<br><font size=2><tt>
Physical state: LinkUp</tt></font>
<br><font size=2><tt>
Rate: 20</tt></font>
<br><font size=2><tt>
Base lid: 2</tt></font>
<br><font size=2><tt>
LMC: 0</tt></font>
<br><font size=2><tt>
SM lid: 2</tt></font>
<br><font size=2><tt>
Capability mask: 0x02510a6a</tt></font>
<br><font size=2><tt>
Port GUID: 0x0002c90200216dc5</tt></font>
<br><font size=2><tt>[koa] (ib) root> ibstatus</tt></font>
<br><font size=2><tt>Infiniband device 'mthca0' port 1 status:</tt></font>
<br><font size=2><tt> default gid:
fe80:0000:0000:0000:0002:c902:0021:6dc5</tt></font>
<br><font size=2><tt> base lid:
0x2</tt></font>
<br><font size=2><tt> sm lid:
0x2</tt></font>
<br><font size=2><tt> state:
3: ARMED</tt></font>
<br><font size=2><tt> phys state:
5: LinkUp</tt></font>
<br><font size=2><tt> rate:
20 Gb/sec (4X DDR)</tt></font>
<br>
<br><font size=2 face="sans-serif">Remote system</font>
<br>
<br><font size=2><tt>[jatoba] (ib) ib> ibstat</tt></font>
<br><font size=2><tt>CA 'mthca0'</tt></font>
<br><font size=2><tt> CA type: MT25204</tt></font>
<br><font size=2><tt> Number of ports: 1</tt></font>
<br><font size=2><tt> Firmware version: 1.0.800</tt></font>
<br><font size=2><tt> Hardware version: a0</tt></font>
<br><font size=2><tt> Node GUID: 0x0002c90200216e40</tt></font>
<br><font size=2><tt> System image GUID: 0x0002c90200216e43</tt></font>
<br><font size=2><tt> Port 1:</tt></font>
<br><font size=2><tt>
State: Initializing</tt></font>
<br><font size=2><tt>
Physical state: LinkUp</tt></font>
<br><font size=2><tt>
Rate: 20</tt></font>
<br><font size=2><tt>
Base lid: 0</tt></font>
<br><font size=2><tt>
LMC: 0</tt></font>
<br><font size=2><tt>
SM lid: 0</tt></font>
<br><font size=2><tt>
Capability mask: 0x02510a68</tt></font>
<br><font size=2><tt>
Port GUID: 0x0002c90200216e41</tt></font>
<br><font size=2><tt>[jatoba] (ib) ib> ibstatus</tt></font>
<br><font size=2><tt>Infiniband device 'mthca0' port 1 status:</tt></font>
<br><font size=2><tt> default gid:
fe80:0000:0000:0000:0002:c902:0021:6e41</tt></font>
<br><font size=2><tt> base lid:
0x0</tt></font>
<br><font size=2><tt> sm lid:
0x0</tt></font>
<br><font size=2><tt> state:
2: INIT</tt></font>
<br><font size=2><tt> phys state:
5: LinkUp</tt></font>
<br><font size=2><tt> rate:
20 Gb/sec (4X DDR)</tt></font>
<br>
<br><font size=2 face="sans-serif">An "ibnetdiscover" on the
local system gives the following:</font>
<br>
<br><font size=2><tt>[koa] (ib) root> ibnetdiscover</tt></font>
<br><font size=2><tt>ibwarn: [20638] handle_port: NodeInfo on DR path [0][1]
port 1 failed, skipping port</tt></font>
<br><font size=2><tt>#</tt></font>
<br><font size=2><tt># Topology file: generated on Fri May 26 14:24:20
2006</tt></font>
<br><font size=2><tt>#</tt></font>
<br><font size=2><tt># Max of 1 hops discovered</tt></font>
<br><font size=2><tt># Initiated from node 0002c90200216dc4 port 0002c90200216dc5</tt></font>
<br>
<br><font size=2><tt>vendid=0x2c9</tt></font>
<br><font size=2><tt>devid=0x6274</tt></font>
<br><font size=2><tt>sysimgguid=0x2c90200216dc7</tt></font>
<br><font size=2><tt>caguid=0x2c90200216dc4</tt></font>
<br><font size=2><tt>Ca 1 "H-0002c90200216dc4"
# koa HCA-1</tt></font>
<br>
<br><font size=2 face="sans-serif">What next, coach?</font>
<br>
<br><font size=2 face="sans-serif"> -Don Albert-</font>