<br><font size=2 face="sans-serif">I was corresponding with Hal Rosenstock
about this problem, but he suggested that I resubmit to a wider audience.
The previous messages are under the subject of "How do
I use "madeye" to diagnose a problem?". I was trying
to use "madeye" to find out if any MAD packets were being received
by a node in which the link fails to initialize.</font>
<br>
<br><font size=2 face="sans-serif">I have a small two-node testbed system
which consists of two EM64T machines ("koa" and "jatoba")
cabled back-to-back with two Mellanox MT25204 (4x DDR) HCAs. This
configuration worked with a backported 2.6.11-34 kernel and revision 6500
from the OpenIB svn trunk. I was able to run basic tests and several
sets of MPI benchmarks.</font>
<br>
<br><font size=2 face="sans-serif">Since moving to a "2.6.16"
kernel and the OFED-1.0 release, we cannot get the link on the "jatoba"
machine to come up. The "madeye" module seems to show
that no MAD packets are being received when the Subnet Manager is run on
the other machine. When I try to run SM on "jatoba", or
try to run any other program that uses MAD, I get process hangs.
Here is a portion of the stack traces for one of the hung processes,
obtained by doing </font><font size=2><tt>"echo t > /proc/sysrq-trigger"</tt></font><font size=2 face="sans-serif">
and looking at the dmesg output.</font>
<br>
<br>
<br><font size=2><tt>ibis D 0000000000000003
0 5489 5097 5522
(NOTLB)</tt></font>
<br><font size=2><tt>ffff8100788c7d28 ffff810037cb9030 ffff8100788c7c78
ffff81007c606640 </tt></font>
<br><font size=2><tt> ffffffff803c1b65 0000000000000001
ffffffff801350ce ffff810003392418 </tt></font>
<br><font size=2><tt> ffff8100788c6000 ffff8100788c7cb8
</tt></font>
<br><font size=2><tt>Call Trace: <ffffffff803c1b65>{_spin_lock_irqsave+14}</tt></font>
<br><font size=2><tt> <ffffffff801350ce>{lock_timer_base+27}
<ffffffff880c4a0d>{:ib_mthca:mthca_table_put+65}</tt></font>
<br><font size=2><tt> <ffffffff803c1c20>{_spin_unlock_irq+9}
<ffffffff803bfd5f>{wait_for_completion+179}</tt></font>
<br><font size=2><tt> <ffffffff80127468>{default_wake_function+0}
<ffffffff80127468>{default_wake_function+0}</tt></font>
<br><font size=2><tt> <ffffffff88023909>{:ib_mad:ib_cancel_rmpp_recvs+144}</tt></font>
<br><font size=2><tt> <ffffffff88020933>{:ib_mad:ib_unregister_mad_agent+1019}</tt></font>
<br><font size=2><tt> <ffffffff8803bc29>{:ib_umad:ib_umad_ioctl+564}
<ffffffff80140025>{autoremove_wake_function+0}</tt></font>
<br><font size=2><tt> <ffffffff80180d4d>{do_ioctl+45}
<ffffffff80181034>{vfs_ioctl+658}</tt></font>
<br><font size=2><tt> <ffffffff8018948e>{mntput_no_expire+28}
<ffffffff80181083>{sys_ioctl+60}</tt></font>
<br><font size=2><tt> <ffffffff8010aa52>{system_call+126}</tt></font>
<br>
<br><font size=2 face="sans-serif">It seems to be a lock or mutex problem,
but I don't know how to proceed from here.</font>
<br>
<br><font size=2 face="sans-serif">Some things I have tried are:</font>
<br>
<ol>
<li value=1><font size=2 face="sans-serif">Connecting the two machines
to a switch instead of back-to-back, to use the SM in the switch.
The link to "koa" comes up, but the link to "jatoba"
does not.<br>
</font>
<li value=2><font size=2 face="sans-serif">Physically swapping the two
HCAs between the two machines: the problem stays on the "jatoba"
side.<br>
</font>
<li value=3><font size=2 face="sans-serif">Turning on "debug_level"
traces with "modprobe ib_mthca debug_level=1" on both machines.
The traces seem to be identical on both, except for the actual PCI
bus location and the memory addresses being mapped. No additional
traces are generated when the hangs occur.</font></ol>
<br><font size=2 face="sans-serif">The machines are both EM64T but are
not identical. The "koa" side has the HCA on PCI "</font><font size=2><tt>06:00.0</tt></font><font size=2 face="sans-serif">",
and the "jatoba" side has the HCA on "</font><font size=2><tt>03:00.0</tt></font><font size=2 face="sans-serif">".
The two machines are:</font>
<br>
<br><font size=2 face="sans-serif"> koa (the working one) is
an Intel SE7520BD2 motherboard (7520 chip set).</font>
<br><font size=2 face="sans-serif"> jatoba (the bad one) is
an Intel SE7525GP2 motherboard (7525 chip set).</font>
<br>
<br><font size=2 face="sans-serif">Can anyone suggest what to try or look
at next?</font>
<br>
<br><font size=2 face="sans-serif"> -Don
Albert-</font>
<br>