[ofa-general] crash in cm_init_qp_rts_attr() - any ideas?

[akepner at sgi.com]

We have a customer who has repeatedly had system panics with 
the following signature:


Unable to handle kernel NULL pointer dereference at 0000000000000010 RIP:
<ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580}
PGD 3a2db6067 PUD 0
Oops: 0000 [1] SMP
last sysfs file: /class/infiniband/mlx4_0/node_guid
CPU 4
Modules linked in: i2c_dev sg sd_mod crc32c libcrc32c iscsi_tcp libiscsi
scsi_transport_iscsi rdma_ucm rdma_cm
iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_umad iw_cxgb3 cxgb3
firmware_class mlx4_ib ib_mthca ib_mad
 ib_core loop numatools xpmem worm mlx4_core libata i2c_i801 scsi_mod i2c_core
shpchp pci_hotplug nfs lockd nfs
_acl af_packet sunrpc e1000
Pid: 3256, comm: star Tainted: G     U 2.6.16.60-0.34-smp #1
RIP: 0010:[<ffffffff882c2c5c>]
<ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580}
RSP: 0018:ffff810369d09d38  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff810419678c00 RCX: 0000000000000008
RDX: 0000000000000246 RSI: ffff810419678d18 RDI: ffff810369d09e70
RBP: ffff810369d09e18 R08: 000000030000003d R09: 0000000000000000
R10: ffff810369d09e18 R11: 0000000000000088 R12: ffff810369d09d88
R13: 0000000000000000 R14: ffff810419678c80 R15: 00000000403500b0
FS:  0000000040354940(0063) GS:ffff810420ffbbc0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000010 CR3: 000000039f0c4000 CR4: 00000000000006e0
Process star (pid: 3256, threadinfo ffff810369d08000, task ffff8103b81b5830)
Stack: ffff810419678a00 ffff810369d09d88 ffff810369d09e18 ffff810369d09e18
       0000000040143430 ffffffff882fb6d5 ffff810376261540 ffff81040bea4740
       ffff810376261540 ffffffff88309285
Call Trace: <ffffffff882fb6d5>{:rdma_cm:rdma_init_qp_attr+209}
       <ffffffff88309285>{:rdma_ucm:ucma_init_qp_attr+160}
       <ffffffff802ea55a>{thread_return+0}
<ffffffff8830832e>{:rdma_ucm:ucma_write+115}
       <ffffffff80186662>{vfs_write+215} <ffffffff80186c2b>{sys_write+69}
      <ffffffff8010adba>{system_call+126}

Code: 8a 40 10 88 85 85 00 00 00 8b 83 38 01 00 00 66 89 45 7a 8a
RIP <ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580} RSP <ffff810369d09d38>


>From a crash dump, I determined that we died in cm_init_qp_rts_attr() 
(it's inline, so it doesn't show up in the traceback) on the line 
labeled below:


static int cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv,
                               struct ib_qp_attr *qp_attr,
                               int *qp_attr_mask)
{
        ........
        if (cm_id_priv->id.lap_state == IB_CM_LAP_UNINIT) {
                .....
        } else {
               *qp_attr_mask = IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE;
               qp_attr->alt_port_num = cm_id_priv->alt_av.port->port_num; <-die




cm_id_priv->alt_av.port is NULL, so it looks as if there's a race 
initializing 'alt_av'.


They are running quite old code (OFED 1.3.1), but I'm not aware of 
anything which would change this behavior in more recent versions, 
though I certainly may have missed something.

Anyone seen similar? Any ideas for a fix, or workaround?

-- 
Arthur