<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>recursion depth exceeded in ipoib_workqueue</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2 FACE="Courier New">environment:</FONT>
<BR><FONT SIZE=2 FACE="Courier New">HCA Port 1 of Host 1 is connected back-to-back to HCA port 1 of Host 2.</FONT>
<BR><FONT SIZE=2 FACE="Courier New">A shell script running on Host 1 loads and unloads the openib driver. On Host 2, the openib driver is up and opensm is running.</FONT></P>
<P><FONT SIZE=2 FACE="Courier New">Host 1: while date ; do</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> /etc/init.d/openibd start</FONT>
<BR> <FONT SIZE=2 FACE="Courier New"> sleep 3</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> /etc/init.d/openibd stop</FONT>
<BR> <FONT SIZE=2 FACE="Courier New"> sleep 1</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> done</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier New"> NOTES: </FONT>
<BR><FONT SIZE=2 FACE="Courier New"> a. sleeps were inserted to give time to opensm on host 2 to respond to changes</FONT>
<BR> <FONT SIZE=2 FACE="Courier New"> b. openibd script attached</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Courier New">Problem -- recursion depth exceeded in ipoib_workqueue:</FONT>
<BR><FONT SIZE=2 FACE="Courier New">/var/log/messages from Host 1</FONT>
<BR><FONT SIZE=2 FACE="Courier New">------------------------------</FONT>
<BR><FONT SIZE=2 FACE="Courier New">ib_mthca: Initializing (0000:04:00.0)</FONT>
<BR><FONT SIZE=2 FACE="Courier New">ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 29 (level, low) -> IRQ 185</FONT>
<BR><FONT SIZE=2 FACE="Courier New">run_workqueue: recursion depth exceeded: 4</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier New">Call Trace:<ffffffff80147a47>{flush_cpu_workqueue+87} <ffffffff803f76d6>{wait_for_completion+230}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff80131d50>{default_wake_function+0} <ffffffff8013fc39>{lock_timer_base+41}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff88078ba3>{:ib_ipoib:ipoib_mcast_stop_thread+99}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff88078cdc>{:ib_ipoib:ipoib_mcast_restart_task+44}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff80147abd>{flush_cpu_workqueue+205} <ffffffff88078cb0>{:ib_ipoib:ipoib_mcast_restart_task+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8013fc39>{lock_timer_base+41} <ffffffff88078ba3>{:ib_ipoib:ipoib_mcast_stop_thread+99}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff88078cdc>{:ib_ipoib:ipoib_mcast_restart_task+44}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff80147abd>{flush_cpu_workqueue+205} <ffffffff88078cb0>{:ib_ipoib:ipoib_mcast_restart_task+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8013fc39>{lock_timer_base+41} <ffffffff88078ba3>{:ib_ipoib:ipoib_mcast_stop_thread+99}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff88078cdc>{:ib_ipoib:ipoib_mcast_restart_task+44}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff88078cb0>{:ib_ipoib:ipoib_mcast_restart_task+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8014795e>{worker_thread+478} <ffffffff80131d50>{default_wake_function+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8012f2d3>{__wake_up_common+67} <ffffffff80131d50>{default_wake_function+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8014beb0>{keventd_create_kthread+0} <ffffffff80147780>{worker_thread+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8014beb0>{keventd_create_kthread+0} <ffffffff8014c009>{kthread+217}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8010e50e>{child_rip+8} <ffffffff8014beb0>{keventd_create_kthread+0}</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> <ffffffff8014bf30>{kthread+0} <ffffffff8010e506>{child_rip+0}</FONT>
</P>
<P><FONT SIZE=2 FACE="Courier New">Please Note:</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> -- Set Multicast List posts the restart task to the ipoib_workqueue (ipoib_main.c:675)</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> -- ipoib_mcast_restart_task (ipoib_multicast.c) calls ipoib_mcast_stop_thread(), which calls flush_workqueue(ipoib_workqueue)</FONT></P>
<P><FONT SIZE=2 FACE="Courier New"> -- so the restart task flushes the work queue its running from.</FONT>
<BR><FONT SIZE=2 FACE="Courier New"> -- Linux prevents the deadlock by testing if the flush is called from the same thread (see linux/workqueue.c:223). If it is, Linux flushes remaining tasks in the work queue (without waiting). This both breaks serialization of tasks in the work queue, and can cause the recursion overflow seen above.</FONT></P>
<P><FONT SIZE=2 FACE="Courier New">Jack</FONT>
</P>
<P><FONT FACE="Arial" SIZE=2 COLOR="#000000"> <<openibd>> </FONT>
</P>
</BODY>
</HTML>