<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: core and ipoib questions and oops</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Problem is at ipoib_multicast.c:223 :</FONT>
<BR> <FONT SIZE=2>if (!memcmp(mcast->mcmember.mgid.raw, priv->dev->broadcast + 4,</FONT>
</P>
<P><FONT SIZE=2>a. r14 contains mcast->dev:</FONT>
<BR><FONT SIZE=2>drivers/infiniband/ulp/ipoib/ipoib_multicast.c:216</FONT>
<BR><FONT SIZE=2> 41a8: 4c 8b b7 f0 00 00 00 mov 0xf0(%rdi),%r14</FONT>
</P>
<P><FONT SIZE=2>NOTE THAT r14 is ZERO. This implies that we still have a pointer to the mcast structure, but</FONT>
<BR><FONT SIZE=2>the entire structure has been zeroed out (the code only sets mcast->dev at mcast struct allocation time -- it never</FONT>
<BR><FONT SIZE=2>zeroes out mcast->dev). This could happen, for example, if mcast was freed, then re-allocated and zeroed.</FONT>
</P>
<P><FONT SIZE=2>b. r13 contains priv (which is obtained via the netdev_priv macro) -- this explains the 0x380 offset from NULL in r13:</FONT>
<BR><FONT SIZE=2>include/linux/netdevice.h:488</FONT>
<BR><FONT SIZE=2> 41b2: 4d 8d ae 80 03 00 00 lea 0x380(%r14),%r13</FONT>
</P>
<P><FONT SIZE=2>We can conclude that the mcast group was deleted, but an mcast completion still got delivered from below by ib_mad</FONT>
<BR><FONT SIZE=2>(the thread which failed was ib_mad1).</FONT>
</P>
<P><FONT SIZE=2>BTW -- this is the same sort of kernel oops that I sent to Roland on 20.9.05 (we also saw 0x0000000000380 in r13 there).</FONT></P>
<P><FONT SIZE=2>This might happen, for example, if when invoking the restart task, wait_for_completion() incorrectly terminated in </FONT>
<BR><FONT SIZE=2>ipoib_mcast_stop_thread() (ipoib_multicast.c:836), then the multicast group was freed ( ipoib_mcast_free() at </FONT>
<BR><FONT SIZE=2>ipoib_multicast.c:915), and finally a callback was invoked after the free.</FONT>
</P>
<P><FONT SIZE=2>Jack</FONT>
</P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Michael S. Tsirkin [<A HREF="mailto:mst@mellanox.co.il">mailto:mst@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Monday, September 26, 2005 4:08 PM</FONT>
<BR><FONT SIZE=2>> To: Tziporet Koren; Jack Morgenstein</FONT>
<BR><FONT SIZE=2>> Subject: Fwd: core and ipoib questions and oops</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Here's an oops I got recently.</FONT>
<BR><FONT SIZE=2>> You might want to look into it.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ----- Forwarded message from "Michael S. Tsirkin" <mst@mellanox.co.il></FONT>
<BR><FONT SIZE=2>> -----</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Subject: core and ipoib questions and oops</FONT>
<BR><FONT SIZE=2>> Date: Mon, 26 Sep 2005 15:49:43 +0300</FONT>
<BR><FONT SIZE=2>> From: "Michael S. Tsirkin" <mst@mellanox.co.il></FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Two questions:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> 1. Roland, looking at ipoib_multicast, I see</FONT>
<BR><FONT SIZE=2>> if (mcast->query) {</FONT>
<BR><FONT SIZE=2>> ib_sa_cancel_query(mcast->query_id,</FONT>
<BR><FONT SIZE=2>> mcast->query);</FONT>
<BR><FONT SIZE=2>> mcast->query = NULL;</FONT>
<BR><FONT SIZE=2>> ipoib_dbg_mcast(priv, "waiting for MGID "</FONT>
<BR><FONT SIZE=2>> IPOIB_GID_FMT "\n",</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> IPOIB_GID_ARG(mcast->mcmember.mgid));</FONT>
<BR><FONT SIZE=2>> wait_for_completion(&mcast->done);</FONT>
<BR><FONT SIZE=2>> }</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> what prevents ipoib_mcast_join_complete from running</FONT>
<BR><FONT SIZE=2>> at the same time and changing mcast->query after we've tested it?</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> 2. All, what happends in the core if I call ib_sa_cancel_query</FONT>
<BR><FONT SIZE=2>> while the completion is running, or has already run?</FONT>
<BR><FONT SIZE=2>> Is it possible that there's a bug that makes it possible for</FONT>
<BR><FONT SIZE=2>> a completion callback to run twice in this case?</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Thanks,</FONT>
<BR><FONT SIZE=2>> MST</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ---</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The following oops happends on svn rev 3535.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> #ifconfig ib0 down</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Unable to handle kernel NULL pointer dereference at 0000000000000388</FONT>
<BR><FONT SIZE=2>> RIP:</FONT>
<BR><FONT SIZE=2>> <ffffffff88045204>{:ib_ipoib:ipoib_mcast_join_finish+100}</FONT>
<BR><FONT SIZE=2>> PGD 172cd4067 PUD 172d16067 PMD 0</FONT>
<BR><FONT SIZE=2>> Oops: 0000 [1] SMP</FONT>
<BR><FONT SIZE=2>> CPU 0</FONT>
<BR><FONT SIZE=2>> Modules linked in: ib_sdp ib_cm ib_ipoib ib_sa ib_umad ib_mthca ib_mad</FONT>
<BR><FONT SIZE=2>> ib_core</FONT>
<BR><FONT SIZE=2>> Pid: 2399, comm: ib_mad1 Not tainted 2.6.13</FONT>
<BR><FONT SIZE=2>> RIP: 0010:[<ffffffff88045204>]</FONT>
<BR><FONT SIZE=2>> <ffffffff88045204>{:ib_ipoib:ipoib_mcast_join_finish+100}</FONT>
<BR><FONT SIZE=2>> RSP: 0018:ffff81017348dc58 EFLAGS: 00010282</FONT>
<BR><FONT SIZE=2>> RAX: 0000000074010000 RBX: 0000000000000000 RCX: 0000000000000010</FONT>
<BR><FONT SIZE=2>> RDX: ffff810177d93380 RSI: ffff810177d93380 RDI: ffff810177d93380</FONT>
<BR><FONT SIZE=2>> RBP: ffff810177d93380 R08: 0000000000000000 R09: ffff81017348dd38</FONT>
<BR><FONT SIZE=2>> R10: ffff81017348ddf8 R11: 0000000000000001 R12: 0000000000000000</FONT>
<BR><FONT SIZE=2>> R13: 0000000000000380 R14: 0000000000000000 R15: ffff810173484898</FONT>
<BR><FONT SIZE=2>> FS: 0000000000000000(0000) GS:ffffffff8064b800(0000)</FONT>
<BR><FONT SIZE=2>> knlGS:0000000000000000</FONT>
<BR><FONT SIZE=2>> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b</FONT>
<BR><FONT SIZE=2>> CR2: 0000000000000388 CR3: 0000000174725000 CR4: 00000000000006e0</FONT>
<BR><FONT SIZE=2>> Process ib_mad1 (pid: 2399, threadinfo ffff81017348c000, task</FONT>
<BR><FONT SIZE=2>> ffff8101773e07f0)</FONT>
<BR><FONT SIZE=2>> Stack: 0000000100000092 0000000000000000 0000000000000096</FONT>
<BR><FONT SIZE=2>> 0000000000000296</FONT>
<BR><FONT SIZE=2>> 0000000000000296 ffffffff8028b8b0 0000000000000096</FONT>
<BR><FONT SIZE=2>> ffff81017d1343c0</FONT>
<BR><FONT SIZE=2>> ffff810172fd50c0 ffff810172daba10</FONT>
<BR><FONT SIZE=2>> Call Trace:<ffffffff8028b8b0>{dma_pool_free+272}</FONT>
<BR><FONT SIZE=2>> <ffffffff88045cbb>{:ib_ipoib:ipoib_mcast_join_complete+43}</FONT>
<BR><FONT SIZE=2>> <ffffffff880021b5>{:ib_core:ib_unpack+198}</FONT>
<BR><FONT SIZE=2>> <ffffffff8803bcc6>{:ib_sa:ib_sa_mcmember_rec_callback+64}</FONT>
<BR><FONT SIZE=2>> <ffffffff8803b49e>{:ib_sa:recv_handler+117}</FONT>
<BR><FONT SIZE=2>> <ffffffff88010e83>{:ib_mad:ib_mad_completion_handler+949}</FONT>
<BR><FONT SIZE=2>> <ffffffff88010ace>{:ib_mad:ib_mad_completion_handler+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff80143489>{worker_thread+478}</FONT>
<BR><FONT SIZE=2>> <ffffffff8012f7da>{default_wake_function+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff8012cff1>{__wake_up_common+64}</FONT>
<BR><FONT SIZE=2>> <ffffffff8012f7da>{default_wake_function+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff801473a8>{keventd_create_kthread+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff801432ab>{worker_thread+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff801473a8>{keventd_create_kthread+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff801474d9>{kthread+204}</FONT>
<BR><FONT SIZE=2>> <ffffffff8010e352>{child_rip+8}</FONT>
<BR><FONT SIZE=2>> <ffffffff801473a8>{keventd_create_kthread+0}</FONT>
<BR><FONT SIZE=2>> <ffffffff8014740d>{kthread+0} <ffffffff8010e34a>{child_rip+0}</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Code: 49 8b 7d 08 48 81 c7 b4 00 00 00 f3 a6 75 17 49 8b 45 70 8b</FONT>
<BR><FONT SIZE=2>> RIP <ffffffff88045204>{:ib_ipoib:ipoib_mcast_join_finish+100} RSP</FONT>
<BR><FONT SIZE=2>> <ffff81017348dc58></FONT>
<BR><FONT SIZE=2>> CR2: 0000000000000388</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> -- </FONT>
<BR><FONT SIZE=2>> MST</FONT>
<BR><FONT SIZE=2>> _______________________________________________</FONT>
<BR><FONT SIZE=2>> openib-general mailing list</FONT>
<BR><FONT SIZE=2>> openib-general@openib.org</FONT>
<BR><FONT SIZE=2>> <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> To unsubscribe, please visit</FONT>
<BR><FONT SIZE=2>> <A HREF="http://openib.org/mailman/listinfo/openib-general" TARGET="_blank">http://openib.org/mailman/listinfo/openib-general</A></FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ----- End forwarded message -----</FONT>
</P>
</BODY>
</HTML>