<br><font size=2 face="sans-serif">Hal,</font>
<br>
<br><font size=2><tt>> Don,<br>
> <br>
> On Wed, 2006-06-21 at 15:46, Don.Albert@Bull.com wrote:<br>
<br>
> > <br>
> > I installed the madeye module on both of the EM64T systems and
checked for<br>
> > packets with dmesg.<br>
> > <br>
> > On the "good" system ("koa"), it shows
lots of packets where the SM<br>
> > is trying to survey the network. On the "bad"
system ("jatoba"), no<br>
> > packets at all are captured. It seems that nothing is being
received<br>
> > on the jatoba side.<br>
> > <br>
> > > We may have gone through this (I don't remember) but can
you try:<br>
> > > 1. Is the firmware version on the node which is not working,
the<br>
> > same as<br>
> > > the one which does ?<br>
> > <br>
> > The firmware version on both MT25204 cards is reported as 1.0.800by<br>
> > the ibstat command.<br>
> > <br>
> > > 2. Are you sure the cable is plugged in properly ? Do you
have<br>
> > another<br>
> > > cable to try ?<br>
> > <br>
> > We have tried switching cables, cabling the HCAs to a switch
instead<br>
> > of back-to-back, and this morning we tried swapping the
HCA cards<br>
> > between the two machines. The problem stays on the "jatoba"<br>
> > machine. There is a difference in the hardware location
on the two<br>
> > machines, in that the PCI bus configuration is different.<br>
> <br>
> Other than that are the 2 machines identical ? Does that include the<br>
> BIOS version ?</tt></font>
<br>
<br><font size=2 face="sans-serif">I don't know the bios version, but the
machines are not identical:</font>
<br>
<br><font size=2 face="sans-serif"> koa (the working one) is
an Intel SE7520BD2 motherboard (7520 chip set).</font>
<br><font size=2 face="sans-serif"> jatoba (the bad one) is
an Intel SE7525GP2 motherboard (7525 chip set).</font>
<br>
<br><font size=2><tt><br>
> <br>
> > The "koa" side has the HCA on PCI "06:00.0",
and the "jatoba" side<br>
> > has the HCA on "03:00.0".<br>
> <br>
> The problem is down at the driver level or lower. Can you enable mthca<br>
> debug_level module parameter (the driver would need to have been build<br>
> with CONFIG_INFINIBAND_MTHCA_DEBUG enabled) and see if anything<br>
> interesting shows up ?<br>
> </tt></font>
<br>
<br><font size=2 face="sans-serif">I finally got a version of mthca built
which allows me to turn on the debug_level. I loaded it on both
machines and looked at the dmesg output. The traces are virtually
identical between the two, except for the PCI address and the memory addresses
for the regions being mapped. When I run a test that gets hung,
there are no additional traces from mthca.</font>
<br><font size=2><tt><br>
> <br>
> > When I try to stop SM on jatoba with the "/etc/init.d/opensmd
stop"<br>
> > script, the script hangs (keeps printing out dots and never<br>
> > terminates) until I break out of it. The OpenSM process
stays hung<br>
> > in execution. Doing a "cat /proc/<pid>/wchan"
shows the OpenSM<br>
> > process waiting in "ib_unregister_mad_agent".<br>
> <br>
> Sounds like a lock or mutex is being held in this case which is blocking<br>
> the unregister. Not sure what it could be but this is a separate issue.<br>
> <br>
</tt></font>
<br><font size=2 face="sans-serif">When I run "ibdiagnet" I
get the following terminal output, and then the terminal stops responding:</font>
<br>
<br><font size=2><tt>[jatoba] (ib) mthca> ibdiagnet</tt></font>
<br><font size=2><tt>Loading IBDIAGNET from: /opt/ofed/lib64/ibdiagnet1.0</tt></font>
<br><font size=2><tt>Loading IBDM from: /opt/ofed/lib64/ibdm1.0</tt></font>
<br><font size=2><tt>-W- Topology file is not specified.</tt></font>
<br><font size=2><tt> Reports regarding cluster links will
use direct routes.</tt></font>
<br><font size=2><tt>-I- Discovering the subnet ... 1 nodes (0 Switches
& 1 CA-s) discovered.</tt></font>
<br><font size=2><tt>-E- Discovery at local link failed: smNodeInfoMad
getByDr 1 - failed 4</tt></font>
<br><font size=2><tt> consecutive times.</tt></font>
<br><font size=2><tt> Exiting.</tt></font>
<br><font size=2><tt><br>
</tt></font><font size=2 face="sans-serif">The "ps axjf" command
(from another terminal) shows the following process tree:</font>
<br>
<br><font size=2><tt>4456 5025 5025 5025 ?
-1 Ss 0 0:00 \_
in.telnetd: albertpc.usnetwork.lan</tt></font>
<br><font size=2><tt>5025 5026 5026 5026 ?
-1 Ss 0 0:00 |
\_ login -- ib</tt></font>
<br><font size=2><tt>5026 5027 5027 5027 pts/0
5489 Ss 500 0:00 |
\_ -bash</tt></font>
<br><font size=2><tt>5027 5095 5095 5027 pts/0
5489 S 0 0:00 |
\_ su</tt></font>
<br><font size=2><tt>5095 5097 5097 5027 pts/0
5489 S 0 0:00 |
\_ bash</tt></font>
<br><font size=2><tt>5097 5489 5489 5027 pts/0
5489 D+ 0 0:00 |
\_ ibis</tt></font>
<br><font size=2><tt>5489 5522 5489 5027 pts/0
5489 S+ 0 0:00 |
\_ ibis</tt></font>
<br><font size=2><tt>5522 5523 5489 5027 pts/0
5489 S+ 0 0:00 |
\_ ibis</tt></font>
<br><font size=2><tt>5522 5524 5489 5027 pts/0
5489 S+ 0 0:00 |
\_ ibis</tt></font>
<br><font size=2><tt>5522 5525 5489 5027 pts/0
5489 S+ 0 0:00 |
\_ ibis</tt></font>
<br>
<br><font size=2 face="sans-serif">The PID 5489 is in an "uninterruptable
sleep" state. Doing a "</font><font size=2><tt>cat
/proc/5489/wchan</tt></font><font size=2 face="sans-serif">" shows
"</font><font size=2><tt>ib_unregister_mad_agent</tt></font><font size=2 face="sans-serif">".
I also did </font><font size=2><tt>"echo t > /proc/sysrq-trigger"</tt></font><font size=2 face="sans-serif">
to get stack traces. Here is the entry for PID 5489:</font>
<br>
<br><font size=2><tt>ibis D 0000000000000003
0 5489 5097 5522
(NOTLB)</tt></font>
<br><font size=2><tt>ffff8100788c7d28 ffff810037cb9030 ffff8100788c7c78
ffff81007c606640 </tt></font>
<br><font size=2><tt> ffffffff803c1b65 0000000000000001
ffffffff801350ce ffff810003392418 </tt></font>
<br><font size=2><tt> ffff8100788c6000 ffff8100788c7cb8
</tt></font>
<br><font size=2><tt>Call Trace: <ffffffff803c1b65>{_spin_lock_irqsave+14}</tt></font>
<br><font size=2><tt> <ffffffff801350ce>{lock_timer_base+27}
<ffffffff880c4a0d>{:ib_mthca:mthca_table_put+65}</tt></font>
<br><font size=2><tt> <ffffffff803c1c20>{_spin_unlock_irq+9}
<ffffffff803bfd5f>{wait_for_completion+179}</tt></font>
<br><font size=2><tt> <ffffffff80127468>{default_wake_function+0}
<ffffffff80127468>{default_wake_function+0}</tt></font>
<br><font size=2><tt> <ffffffff88023909>{:ib_mad:ib_cancel_rmpp_recvs+144}</tt></font>
<br><font size=2><tt> <ffffffff88020933>{:ib_mad:ib_unregister_mad_agent+1019}</tt></font>
<br><font size=2><tt> <ffffffff8803bc29>{:ib_umad:ib_umad_ioctl+564}
<ffffffff80140025>{autoremove_wake_function+0}</tt></font>
<br><font size=2><tt> <ffffffff80180d4d>{do_ioctl+45}
<ffffffff80181034>{vfs_ioctl+658}</tt></font>
<br><font size=2><tt> <ffffffff8018948e>{mntput_no_expire+28}
<ffffffff80181083>{sys_ioctl+60}</tt></font>
<br><font size=2><tt> <ffffffff8010aa52>{system_call+126}</tt></font>
<br>
<br><font size=2 face="sans-serif">I think I agree that this is a lock
or mutex problem. How can I determine which lock or mutex is being
held, resulting in the hang?</font>
<br>
<br><font size=2 face="sans-serif"> -Don
Albert-</font>
<br>