[openib-general] OOPS: ib_mad crashery on bootup

[Sean Hefty]

I made several modifications to the MAD layer to assist with debugging this, 
and after a multitude of test runs I was able to see the output shown below.

Basically, a send work request is completing, but the MAD associated with 
the request has been freed/corrupted.  It's likely that the errors for MADs 
"not at the head of send queue" are a result of the corrupted MAD.

- Sean

cmpost: starting client
cmpost: connecting
cmpost: connect time: 4072000 us
cmpost: waiting to disconnect
cmpost: test complete
send_comp_handler - completion for MAD not on send queue QP: 1
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not on send queue QP: 1
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not on send queue QP: 1
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
send_comp_handler - completion for MAD not at head of send queue QP: 1
found later in queue
Slab corruption: start=dfc9ddd8, len=256
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01305f8>](worker_thread+0x1a8/0x230)
000: 60 d3 c9 df 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Prev obj: start=dfc9dccc, len=256
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<f8c6c6eb>](ib_mad_send_done_handler+0x2b/0x120 [ib_mad])
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Next obj: start=dfc9dee4, len=256
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<f8c6c6eb>](ib_mad_send_done_handler+0x2b/0x120 [ib_mad])
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b