<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] Problem with 2.4.24 and gen1</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Hi,</FONT>
</P>
<P><FONT SIZE=2>The problem is that the driver does not get the interrupt for the command completion, </FONT>
<BR><FONT SIZE=2>and thus you get the error: "Command not completed after timeout".</FONT>
</P>
<P><FONT SIZE=2>It is related to the OS & system you are using. What is the distribution you are using? We once saw such problems with older versions of SuSE.</FONT></P>
<P><FONT SIZE=2>Try to add append="acpi=off" to the lilo you are using or add also disableapic in the same append line.</FONT>
</P>
<BR>
<P><FONT SIZE=2>Tziporet</FONT>
</P>
<BR>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Ken MacInnis [<A HREF="mailto:kcm@psc.edu">mailto:kcm@psc.edu</A>]</FONT>
<BR><FONT SIZE=2>Sent: Sunday, October 31, 2004 8:20 PM</FONT>
<BR><FONT SIZE=2>To: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>Subject: [openib-general] Problem with 2.4.24 and gen1</FONT>
</P>
<BR>
<P><FONT SIZE=2>Hi,</FONT>
</P>
<P><FONT SIZE=2>I've got a fairly modified kernel here I'm trying to get a OpenIB stack</FONT>
<BR><FONT SIZE=2>running on. It's a vanilla 2.4.24 kernel with Lustre and other patches</FONT>
<BR><FONT SIZE=2>in it, but I'm seeing this when I modprobe ib_tavor:</FONT>
</P>
<P><FONT SIZE=2>Oct 31 13:13:05 samwise kernel: THH(1): cmdif.c[1190]: Command not</FONT>
<BR><FONT SIZE=2>completed after timeout: cmd=TAV</FONT>
<BR><FONT SIZE=2>OR_IF_CMD_MAD_IFC (0x24), token=0x1400, pid=0x8E1, go=0</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:05 samwise kernel: THH(1): CMD ERROR DUMP. opcode=0x24,</FONT>
<BR><FONT SIZE=2>opc_mod = 0x1, exec_time_micro</FONT>
<BR><FONT SIZE=2>=300000000</FONT>
<BR><FONT SIZE=2>.</FONT>
<BR><FONT SIZE=2>.</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): cmdif.c[842]: Failed command</FONT>
<BR><FONT SIZE=2>0x24 (TAVOR_IF_CMD_MAD_IFC): s</FONT>
<BR><FONT SIZE=2>tatus=0x103 (0x0103 - unexpected error - fatal)</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel:</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): thh_hob.c[2790]:</FONT>
<BR><FONT SIZE=2>THH_hob_query_port_prop: cmdif returned FA</FONT>
<BR><FONT SIZE=2>TAL</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: VIPKL(1): qpm.c[278]: QPM_new:</FONT>
<BR><FONT SIZE=2>HOBKL_query_port_prop returned with</FONT>
<BR><FONT SIZE=2>error: -254 = VAPI_EFATAL</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: VIPKL(1): qpm.c[302]: QPM_new:</FONT>
<BR><FONT SIZE=2>returned with error: -254 = VAPI_EF</FONT>
<BR><FONT SIZE=2>ATAL</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): thh_hob.c[3474]:</FONT>
<BR><FONT SIZE=2>THH_hob_fatal_err_thread: RECEIVED FATAL E</FONT>
<BR><FONT SIZE=2>RROR WAKEUP</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): thh_hob.c[4490]:</FONT>
<BR><FONT SIZE=2>THH_hob_halt_hca: HALT HCA returned 0x103</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): thh_hob.c[1620]:</FONT>
<BR><FONT SIZE=2>THH_hob_destroy: FATAL ERROR</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: THH(1): thh_hob.c[1627]:</FONT>
<BR><FONT SIZE=2>THH_hob_destroy: PERFORMING SW RESET. pa=0</FONT>
<BR><FONT SIZE=2>xFE9F0010 va=0xF8A01010</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel:</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel: Mellanox Tavor Device Driver is creating</FONT>
<BR><FONT SIZE=2>device "InfiniHost0" (bus=0</FONT>
<BR><FONT SIZE=2>4, devfn=00)</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel:</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel:</FONT>
<BR><FONT SIZE=2>[KERNEL_IB][_tsIbTavorInitOne][tavor_main.c:86]InfiniHost0: VAPI_ope</FONT>
<BR><FONT SIZE=2>n_hca failed, status -254 (Fatal error (Local Catastrophic Error))</FONT>
<BR><FONT SIZE=2>Oct 31 13:13:06 samwise kernel:</FONT>
<BR><FONT SIZE=2>[SRPTP][srp_host_init][srp_host.c:1495]SRP Host using indirect addre</FONT>
<BR><FONT SIZE=2>ssing</FONT>
</P>
<BR>
<P><FONT SIZE=2>This occurs with an older openib rev (200-ish) as well as one up-to-date</FONT>
<BR><FONT SIZE=2>as of today.</FONT>
</P>
<P><FONT SIZE=2>Everything else (modules.conf, etc.) is set up as it has been when I was</FONT>
<BR><FONT SIZE=2>messing with 2.4 kernels and OpenIB a few months ago, so I'm not</FONT>
<BR><FONT SIZE=2>thinking it's related to such.</FONT>
</P>
<P><FONT SIZE=2>Any ideas? Yes, I know it's 2.4 as well as a fairly older 2.4, but I</FONT>
<BR><FONT SIZE=2>have no choice here. :) lspci -vvv bits follow.</FONT>
</P>
<P><FONT SIZE=2>03:01.0 PCI bridge: Mellanox Technology: Unknown device 5a46 (rev a1)</FONT>
<BR><FONT SIZE=2>(prog-if 00 [Normal decode])</FONT>
<BR><FONT SIZE=2> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-</FONT>
<BR><FONT SIZE=2>ParErr- Stepping- SERR+ FastB2B-</FONT>
<BR><FONT SIZE=2> Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT>
<BR><FONT SIZE=2><TAbort- <MAbort- >SERR- <P</FONT>
<BR><FONT SIZE=2>ERR-</FONT>
<BR><FONT SIZE=2> Latency: 64, cache line size 10</FONT>
<BR><FONT SIZE=2> Bus: primary=03, secondary=04, subordinate=04, sec-latency=64</FONT>
<BR><FONT SIZE=2> I/O behind bridge: 0000f000-00000fff</FONT>
<BR><FONT SIZE=2> Memory behind bridge: fe700000-fe9fffff</FONT>
<BR><FONT SIZE=2> Prefetchable memory behind bridge: </FONT>
<BR><FONT SIZE=2>00000000eb200000-00000000fc200000</FONT>
<BR><FONT SIZE=2> BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-</FONT>
<BR><FONT SIZE=2> Capabilities: [70] PCI-X non-bridge device.</FONT>
<BR><FONT SIZE=2> Command: DPERE+ ERO+ RBC=0 OST=4</FONT>
<BR><FONT SIZE=2> Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,</FONT>
<BR><FONT SIZE=2>DC=simple, DMMRBC=0, DMOST=0, D</FONT>
<BR><FONT SIZE=2>MCRS=0, RSCEM-</FONT>
<BR><FONT SIZE=2>04:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1)</FONT>
<BR><FONT SIZE=2> Subsystem: Mellanox Technology: Unknown device 5a44</FONT>
<BR><FONT SIZE=2> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-</FONT>
<BR><FONT SIZE=2>ParErr- Stepping- SERR+ FastB2B-</FONT>
<BR><FONT SIZE=2> Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT>
<BR><FONT SIZE=2><TAbort- <MAbort- >SERR- <P</FONT>
<BR><FONT SIZE=2>ERR-</FONT>
<BR><FONT SIZE=2> Latency: 64, cache line size 10</FONT>
<BR><FONT SIZE=2> Interrupt: pin A routed to IRQ 25</FONT>
<BR><FONT SIZE=2> Region 0: Memory at fe900000 (64-bit, non-prefetchable) [size=1M]</FONT>
<BR><FONT SIZE=2> Region 2: Memory at fb800000 (64-bit, prefetchable) [size=8M]</FONT>
<BR><FONT SIZE=2> Region 4: Memory at f0000000 (64-bit, prefetchable) [size=128M]</FONT>
<BR><FONT SIZE=2> Capabilities: [40] #11 [001f]</FONT>
<BR><FONT SIZE=2> Capabilities: [60] Message Signalled Interrupts: 64bit+</FONT>
<BR><FONT SIZE=2>Queue=0/5 Enable-</FONT>
<BR><FONT SIZE=2> Address: 0000000000000000 Data: 0000</FONT>
<BR><FONT SIZE=2> Capabilities: [70] PCI-X non-bridge device.</FONT>
<BR><FONT SIZE=2> Command: DPERE- ERO- RBC=3 OST=1</FONT>
<BR><FONT SIZE=2> Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,</FONT>
<BR><FONT SIZE=2>DC=simple, DMMRBC=0, DMOST=0, D</FONT>
<BR><FONT SIZE=2>MCRS=0, RSCEM-</FONT>
</P>
<BR>
<P><FONT SIZE=2>Ken</FONT>
</P>
<P><FONT SIZE=2>-- </FONT>
<BR><FONT SIZE=2>Ken MacInnis - Systems Engineer, PSC - <A HREF="http://www.psc.edu/~kcm/" TARGET="_blank">http://www.psc.edu/~kcm/</A></FONT>
<BR><FONT SIZE=2>kcm at psc dot edu - +1 412 268 9833 (w) - +1 412 268 5832 (f)</FONT>
<BR><FONT SIZE=2>Pittsburgh Supercomputing Center - 4400 Fifth Ave - Pittsburgh, PA 15213</FONT>
<BR><FONT SIZE=2>_______________________________________________</FONT>
<BR><FONT SIZE=2>openib-general mailing list</FONT>
<BR><FONT SIZE=2>openib-general@openib.org</FONT>
<BR><FONT SIZE=2><A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
</P>
<P><FONT SIZE=2>To unsubscribe, please visit <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
</P>
</BODY>
</HTML>