[ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup()

[akepner at sgi.com]

On Wed, May 20, 2009 at 10:28:38AM -0700, Roland Dreier wrote:
> 
>  > We've seen a few instances of a crash in ipoib_neigh_cleanup() due to 
>  > the use of a stale pointer:
>  > 
>  > 
>  > 848         neigh = *to_ipoib_neigh(n); <- read neigh (no locking)
>  > .....
>  > 858         spin_lock_irqsave(&priv->lock, flags);
>  > 859
>  > 860         if (neigh->ah) <--- at this point neigh may be stale
>  > 861                 ah = neigh->ah;
>  > 862         list_del(&neigh->list);
>  > 863         ipoib_neigh_free(n->dev, neigh);
>  > 864
>  > 865         spin_unlock_irqrestore(&priv->lock, flags);
> 
> I'd like to understand the bug first -- how is the neighbour being
> destroyed out from under us in ipoib_neigh_cleanup()?  I would have
> thought the cleanup function would run when no references to the struct
> remain but before it's freed.
> 

I should have been more specific - it's not the ipoib_neigh structure 
pointer itself, but the list inside the structure where we've found a 
problem. The specific crash we are trying to fix is when someone does 
list_del(&neigh->list) just before we acquire the lock at line 860. 
(But all the callers of list_del(&neigh->list) subsequently call 
ipoib_neigh_free(), too, so the neigh pointer is bad.)

The signature of the crash is like this:

Unable to handle kernel paging request at 0000000000100108 RIP: ^M
                                          ^^^^^^^^^^^^^^^^
                                          LIST_POISON1+0x8
<ffffffff882a3794>{:ib_ipoib:ipoib_neigh_cleanup+368}^M
PGD 4152b3067 PUD 413ee4067 PMD 0 ^M
Oops: 0002 [1] SMP ^M
last sysfs file: /class/infiniband/mthca1/node_type^M
CPU 7 ^M
Modules linked in: sg sd_mod crc32c libcrc32c rdma_ucm rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_umad mlx4_ib mlx4_core ib_mthca ib_mad ib_core iscsi_tcp libiscsi scsi_transport_iscsi loop numatools xpmem i2c_i801 libata i2c_core scsi_mod shpchp pci_hotplug nfs lockd nfs_acl af_packet sunrpc e1000^M
Pid: 0, comm: swapper Tainted: G     U 2.6.16.54-0.2.5-smp #1^M
RIP: 0010:[<ffffffff882a3794>] <ffffffff882a3794>{:ib_ipoib:ipoib_neigh_cleanup+368}^M
RSP: 0018:ffff81042088bea8  EFLAGS: 00010082^M
RAX: 0000000000200200 RBX: ffff8104162fdd40 RCX: ffff8104162fdd98^M
RDX: 0000000000100100 RSI: ffff8104162fdd40 RDI: ffff81041b2f8500^M
RBP: ffff8103c7600480 R08: ffff81041e7b10f0 R09: 0000000000000000^M
R10: ffff810420885e48 R11: 0000000000003a98 R12: ffff81041aa39480^M
R13: ffff81041b2f8500 R14: 0000000000000246 R15: ffffffff803d3ff0^M
FS:  0000000000000000(0000) GS:ffff810420fdc2c0(0000) knlGS:0000000000000000^M
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b^M
CR2: 0000000000100108 CR3: 0000000417469000 CR4: 00000000000006e0^M
Process swapper (pid: 0, threadinfo ffff810420884000, task ffff81042083f100)^M
Stack: ffff8103c7600480 0000000000000000 ffff8103c7600480 0000000108a8ec53 ^M
       ffffffff803c2700 ffffffff80284ebd ffff8103c7600480 ffffffff803862a0 ^M
       ffff81041610c380 ffffffff802871ca ^M
Call Trace: <IRQ> <ffffffff80284ebd>{neigh_destroy+197}^M
       <ffffffff802871ca>{neigh_periodic_timer+249} <ffffffff802870d1>{neigh_periodic_timer+0}^M
       <ffffffff8013ba84>{run_timer_softirq+348} <ffffffff801377d1>{__do_softirq+85}^M
       <ffffffff8010c11e>{call_softirq+30} <ffffffff8010d07c>{do_softirq+44}^M
       <ffffffff80109e3a>{mwait_idle+0} <ffffffff8010ba78>{apic_timer_interrupt+132} <EOI>^M
       <ffffffff80109e3a>{mwait_idle+0} <ffffffff80109e70>{mwait_idle+54}^M
       <ffffffff80109e17>{cpu_idle+151} <ffffffff80119036>{start_secondary+1240}^M

-- 
Arthur