<br><font size=2 face="sans-serif">Michael,</font>
<br>
<br><font size=2><tt>> > The ib_mthca module now initializes
correctly on<br>
> > both EM64T machines. I noticed some discussion between
you and Roland about<br>
> > making the parameter "fw_cmd_doorbell=0" the default.
Did this <br>
> > occur in RC5?<br>
> <br>
> Yes, we changed fw_cmd_doorbell to 0 by default for now because it
seemed<br>
> safer. I expect if you load mthca with fw_cmd_doorbell=1 you still
get an<br>
> error, isn't that right?<br>
> <br>
</tt></font>
<br><font size=2><tt>Although the change in RC5 for fw_cmd_doorbell *seemed*
to allow the ib_mthca module to initialize, I don't think I am out of the
woods yet on this particular machine. The link never comes up, and
the other machine, which is connected back to back with this one, and on
which I am trying to run OpenSM, does not get a response to its MAD
packets. When I try to shut down the openib stack with the "/etc/init.d/openibd
stop" script, the processes hang trying to set device "ib0"
down. Here is an excerpt from a terminal session:</tt></font>
<br>
<br><font size=2><tt> [jatoba] (ib) ib> ibstat</tt></font>
<br><font size=2><tt> CA 'mthca0'</tt></font>
<br><font size=2><tt> CA type: MT25204</tt></font>
<br><font size=2><tt> Number of ports: 1</tt></font>
<br><font size=2><tt> Firmware version: 1.0.800</tt></font>
<br><font size=2><tt> Hardware version: a0</tt></font>
<br><font size=2><tt> Node GUID: 0x0002c90200216e40</tt></font>
<br><font size=2><tt> System image GUID: 0x0002c90200216e43</tt></font>
<br><font size=2><tt> Port 1:</tt></font>
<br><font size=2><tt> State: Initializing</tt></font>
<br><font size=2><tt> Physical state: LinkUp</tt></font>
<br><font size=2><tt> Rate: 20</tt></font>
<br><font size=2><tt> Base lid: 0</tt></font>
<br><font size=2><tt> LMC: 0</tt></font>
<br><font size=2><tt> SM lid: 0</tt></font>
<br><font size=2><tt> Capability mask: 0x02510a68</tt></font>
<br><font size=2><tt> Port GUID: 0x0002c90200216e41</tt></font>
<br><font size=2><tt> [jatoba] (ib) ib> ibstatus</tt></font>
<br><font size=2><tt> Infiniband device 'mthca0' port 1 status:</tt></font>
<br><font size=2><tt> default gid: fe80:0000:0000:0000:0002:c902:0021:6e41</tt></font>
<br><font size=2><tt> base lid: 0x0</tt></font>
<br><font size=2><tt> sm lid:
0x0</tt></font>
<br><font size=2><tt> state:
2: INIT</tt></font>
<br><font size=2><tt> phys state: 5: LinkUp</tt></font>
<br><font size=2><tt> rate:
20 Gb/sec (4X DDR)</tt></font>
<br>
<br><font size=2><tt> [jatoba] (ib) ib> /etc/init.d/opensmd
status</tt></font>
<br><font size=2><tt> opensm is stopped</tt></font>
<br><font size=2><tt> [jatoba] (ib) ib> /etc/init.d/openibd
status</tt></font>
<br>
<br><font size=2><tt> HCA driver loaded</tt></font>
<br>
<br><font size=2><tt> Configured devices:</tt></font>
<br><font size=2><tt> ib0</tt></font>
<br>
<br><font size=2><tt> Currently active devices:</tt></font>
<br><font size=2><tt> ib0</tt></font>
<br>
<br><font size=2><tt> The following modules are also loaded:</tt></font>
<br>
<br><font size=2><tt> ib_cm</tt></font>
<br>
<br><font size=2><tt> [jatoba] (ib) ib> /etc/init.d/openibd
stop</tt></font>
<br>
<br>
<br><font size=2><tt>At this point the command hangs. Doing a "ps
-ef" from another terminal reveals:</tt></font>
<br>
<br><font size=2><tt> root 6882 6755
0 15:31 pts/0 00:00:00 /bin/bash /etc/init.d/openibd
stop</tt></font>
<br><font size=2><tt> root 7012 6882
0 15:31 pts/0 00:00:00 /bin/bash /sbin/ifdown ib0</tt></font>
<br><font size=2><tt> root 7031 7012
0 15:31 pts/0 00:00:00 ip link set dev ib0 down</tt></font>
<br>
<br><font size=2><tt>I tried using gdb to "attach" to process
7031 to see its stack, but that hung too, as well as an attempt to see
what the status of the interface was with "/sbin/ifconfig".
</tt></font>
<br>
<br><font size=2><tt>It is rather difficult for me to debug this sort of
hang, since I telecommute from Tucson and the machines are located in Phoenix.
Anyone have any suggestions?</tt></font>
<br>
<br><font size=2><tt> -Don Albert-</tt></font>
<br>