[ofa-general] [Bug 465] IPoIB CM HA fails after several hours of failures

Tue Mar 27 02:01:36 PDT 2007

Pls do not reply to this message.
I am copying the general list on this bug report so that
we can start discussion by mail.
I am then going to reply copying the bugzilla reflector
so that "reply all" will get tracked in bugzilla.

> I've been trying IPoIB CM HA for a few weeks, and can't get it to run
> overnight.  I've tried both SLES10 (LionCub DDR) and RHEL4 (LionMini SDR and
> LionMini DDR).
> 
> I run netperf 2.4.1 with large socket buffers:
> 
> netperf241 -H 192.168.2.46 -D -l 36000 --  -s 349520 -S 349520 -m 65536
> 
> While netperf is running, I start flipping IB ports once every 10 seconds.
> 
> After a few hours, I sometimes see netperf throughput drop to almost zero:
> 
> ...
> Interim result:    0.45 10^6bits/s over 13.91 seconds
> Interim result:    0.68 10^6bits/s over 6.91 seconds
> Interim result:    0.06 10^6bits/s over 112.48 seconds
> Interim result:    0.10 10^6bits/s over 60.32 seconds
> Interim result:    0.43 10^6bits/s over 14.55 seconds
> 
> Other times netperf hangs or fails.
> 
> Restarting netperf as is never works.  Sometimes I can restart netperf with
> default socket buffer sizes.
> 
> ----- End forwarded message -----

Here is the dmesg output.
A copy is here:
https://bugs.openfabrics.org/attachment.cgi?id=106&action=view

ib0: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:0849 MTU > 0
ib1: Send unicast ARP to 0005
ib0: PathRec LID 0x0005 for GID fe80:0000:0000:0000:0005:ad00:0020:0849
ib0: Created ah 00000101b3983a00
ib0: created address handle 00000101be579c80 for LID 0x0005, SL 0
ib0: Send unicast ARP to 0005
ib1: REQ arrived
ib1: REQ arrived
ib1: failed cm send event (status=12, wrid=28 vend_err 81)
ib1: Destroy active connection 0x4c05bd head 0x5791d tail 0x5791d
ib1: Request connection 0x4e05bd for gid fe80:0000:0000:0000:0005:ad00:0020:0849 qpn 0x404
ib1: REP received.
ib0: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib0: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib0: Created ah 00000101b39835c0
ib0: created address handle 00000101bbff2840 for LID 0x0007, SL 0
ib0: Send unicast ARP to 0007
ib1: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib1: Created ah 00000101a3816340
ib1: created address handle 00000101b3983540 for LID 0x0007, SL 0
ib1: Send unicast ARP to 0007
ib0: REQ arrived
ib1: failed cm send event (status=12, wrid=21 vend_err 81)
ib1: Destroy active connection 0x4e05bd head 0xdcd8 tail 0xdcd6
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101bfc47b00
ib0: Created ah 00000101be977540
ib0: Created ah 00000101aa5e0f00
ib0: Created ah 00000101a3816dc0
ib0: Created ah 00000101ab66c980
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib0: enabling connected mode will cause multicast packet drops
ib1: PathRec status -110 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib0: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib0: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib0: Created ah 00000101bbff2a00
ib0: created address handle 00000101bbff2200 for LID 0x0007, SL 0
ib0: Request connection 0x4f05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x4f05bd head 0x5 tail 0x3
ib0: Request connection 0x5005bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x5005bd head 0x3 tail 0x3
ib0: Request connection 0x5105bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x5105bd head 0x2 tail 0x2
ib0: Request connection 0x5205bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x5205bd head 0x3 tail 0x3
ib0: Request connection 0x5305bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x5305bd head 0x4 tail 0x4
ib0: Request connection 0x5405bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x5405bd head 0x4 tail 0x4
ib0: Request connection 0x5505bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x5505bd head 0x2 tail 0x2
ib0: Request connection 0x5605bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x5605bd head 0x4 tail 0x4
ib0: Request connection 0x5705bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x5705bd head 0x3 tail 0x3
ib0: Request connection 0x5805bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x5805bd head 0x2 tail 0x2
ib0: Request connection 0x5905bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x5905bd head 0x4 tail 0x4
ib0: Request connection 0x5a05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x5a05bd head 0x3 tail 0x3
ib0: Request connection 0x5b05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x5b05bd head 0x2 tail 0x2
ib0: Request connection 0x5c05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x5c05bd head 0x4 tail 0x4
ib0: Request connection 0x5d05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x5d05bd head 0x3 tail 0x3
ib0: Request connection 0x5e05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x5e05bd head 0x2 tail 0x2
ib0: Request connection 0x5f05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x5f05bd head 0x6 tail 0x6
ib0: Request connection 0x6005bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6005bd head 0x2 tail 0x2
ib0: Request connection 0x6105bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x6105bd head 0x4 tail 0x4
ib0: Request connection 0x6205bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x6205bd head 0x3 tail 0x3
ib0: Request connection 0x6305bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6305bd head 0x2 tail 0x2
ib0: Request connection 0x6405bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x6405bd head 0x6 tail 0x6
ib0: Request connection 0x6505bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6505bd head 0x2 tail 0x2
ib0: Request connection 0x6605bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x6605bd head 0x4 tail 0x4
ib0: Request connection 0x6705bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x6705bd head 0x3 tail 0x3
ib0: Request connection 0x6805bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6805bd head 0x2 tail 0x2
ib0: Request connection 0x6905bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x6905bd head 0x6 tail 0x6
ib0: Request connection 0x6a05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6a05bd head 0x2 tail 0x2
ib0: Request connection 0x6b05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x6b05bd head 0x6 tail 0x6
ib0: Request connection 0x6c05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6c05bd head 0x2 tail 0x2
ib0: Request connection 0x6d05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x6d05bd head 0x4 tail 0x4
ib0: Request connection 0x6e05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x6e05bd head 0x3 tail 0x3
ib0: Request connection 0x6f05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x6f05bd head 0x2 tail 0x2
ib0: Request connection 0x7005bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7005bd head 0x4 tail 0x4
ib0: Request connection 0x7105bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x7105bd head 0x3 tail 0x3
ib0: Request connection 0x7205bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x7205bd head 0x2 tail 0x2
ib0: Request connection 0x7305bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7305bd head 0x4 tail 0x4
ib0: Request connection 0x7405bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x7405bd head 0x3 tail 0x3
ib0: Request connection 0x7505bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x7505bd head 0x2 tail 0x2
ib0: Request connection 0x7605bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7605bd head 0x4 tail 0x4
ib0: Request connection 0x7705bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x7705bd head 0x3 tail 0x3
ib0: Request connection 0x7805bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x7805bd head 0x2 tail 0x2
ib0: Request connection 0x7905bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7905bd head 0x4 tail 0x4
ib0: Request connection 0x7a05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x7a05bd head 0x3 tail 0x3
ib0: Request connection 0x7b05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x7b05bd head 0x2 tail 0x2
ib0: Request connection 0x7c05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7c05bd head 0x4 tail 0x4
ib0: Request connection 0x7d05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x7d05bd head 0x3 tail 0x3
ib0: Request connection 0x7e05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x7e05bd head 0x2 tail 0x2
ib0: Request connection 0x7f05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x7f05bd head 0x4 tail 0x4
ib0: Request connection 0x8005bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x8005bd head 0x3 tail 0x3
ib0: Request connection 0x8105bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x8105bd head 0x2 tail 0x2
ib0: Request connection 0x8205bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x8205bd head 0x6 tail 0x6
ib0: Request connection 0x8305bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x8305bd head 0x2 tail 0x2
ib0: Request connection 0x8405bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=5 vend_err 87)
ib0: Destroy active connection 0x8405bd head 0x6 tail 0x6
ib0: Request connection 0x8505bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=1 vend_err 87)
ib0: Destroy active connection 0x8505bd head 0x2 tail 0x2
ib0: Request connection 0x8605bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=3 vend_err 87)
ib0: Destroy active connection 0x8605bd head 0x4 tail 0x4
ib0: Request connection 0x8705bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:084a
ib1: flushing
ib1: downing ib_dev
ib0: flushing
ib0: downing ib_dev
ib0: Destroy active connection 0x8705bd head 0x0 tail 0x0
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101be888b00
ib0: Created ah 00000101b35f4240
ib0: Created ah 00000101b2543f40
ib0: Created ah 00000101a3816fc0
ib0: Created ah 00000101b25433c0
ib0: enabling connected mode will cause multicast packet drops
ib1: Created ah 00000101a7d470c0
ib1: Created ah 00000101a7d47e00
ib1: Created ah 00000101a7d47700
ib1: Created ah 00000101a7d471c0
ib1: Created ah 00000101bbff2a00
ib0: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib0: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib0: Created ah 00000101baf75580
ib0: created address handle 00000101a3816bc0 for LID 0x0007, SL 0
ib0: Request connection 0x8805bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: REP received.
ib0: failed cm send event (status=13, wrid=2 vend_err 87)
ib0: Destroy active connection 0x8805bd head 0x3 tail 0x3
ib0: Request connection 0x8905bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib0: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:084a
ib1: flushing
ib1: downing ib_dev
ib0: Destroy active connection 0x8905bd head 0x0 tail 0x0
ib1: Created ah 00000101a3816e80
ib1: Created ah 00000101a7d47fc0
ib1: Created ah 00000101bfc0d700
ib1: Created ah 00000101a7d47600
ib1: Created ah 00000101a3816400
ib1: enabling connected mode will cause multicast packet drops
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:0849 MTU > 0
ib1: PathRec LID 0x0005 for GID fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Created ah 00000101be888b00
ib1: created address handle 00000101b25430c0 for LID 0x0005, SL 0
ib1: Request connection 0x8a05bd for gid fe80:0000:0000:0000:0005:ad00:0020:0849 qpn 0x404
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Destroy active connection 0x8a05bd head 0x0 tail 0x0
ib1: Created ah 00000101abe60e80
ib1: Created ah 00000101b2d5f540
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101b2543740
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101bdcaa7c0
ib1: Created ah 00000101abe608c0
ib1: Created ah 00000101bf443240
ib1: Created ah 00000101bf443f40
ib1: Created ah 00000101be401b40
ib1: enabling connected mode will cause multicast packet drops
ib0: Created ah 00000101bfc40780
ib0: Created ah 00000101bfc475c0
ib0: Created ah 00000101ab66cdc0
ib0: Created ah 00000101bfc40c40
ib0: Created ah 00000101be977c00
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:0849 MTU > 0
ib1: PathRec LID 0x0005 for GID fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Created ah 00000101bfc47340
ib1: created address handle 00000101bfc814c0 for LID 0x0005, SL 0
ib1: Request connection 0x8b05bd for gid fe80:0000:0000:0000:0005:ad00:0020:0849 qpn 0x404
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Destroy active connection 0x8b05bd head 0x0 tail 0x0
ib0: Created ah 00000101b38a9480
ib0: Created ah 00000101bfc812c0
ib0: Created ah 00000101be401300
ib0: Created ah 00000101b38a9740
ib0: Created ah 00000101bfc40b00
ib0: enabling connected mode will cause multicast packet drops
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101a3816680
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101b38a9f40
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101b38a9640
ib0: Created ah 00000101b38a9ec0
ib0: Created ah 00000101bdfbb7c0
ib0: Created ah 00000101bfc40780
ib0: Created ah 00000101abe60e80
ib0: enabling connected mode will cause multicast packet drops
ib1: Created ah 00000101b38a98c0
ib1: Created ah 00000101bfc40a00
ib1: Created ah 00000101b25430c0
ib1: Created ah 00000101bfc4b1c0
ib1: Created ah 00000101ab66ce80
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 0000010037dbd840
ib1: Created ah 00000101b2543d40
ib1: Created ah 00000101be977c00
ib1: Created ah 00000101ba22c1c0
ib1: Created ah 00000101bdfbbdc0
ib1: enabling connected mode will cause multicast packet drops
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101beb8b600
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101bff30c00
ib1: Created ah 00000101aa5e0c40
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101beb8b780
ib1: Created ah 0000010037dbd5c0
ib1: Created ah 00000101aa5e0140
ib1: Created ah 00000101b2ad71c0
ib1: Created ah 00000101beb8b300
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101ab19f940
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101a3816500
ib1: Created ah 00000101a3816ec0
ib1: Created ah 00000101a3816d00
ib1: Created ah 00000101b28661c0
ib1: Created ah 00000101b2866380
ib1: enabling connected mode will cause multicast packet drops
ib0: Created ah 00000101bff30240
ib0: Created ah 00000101a3816700
ib0: Created ah 00000101beb8b400
ib0: Created ah 00000101b2866c80
ib0: Created ah 00000101aa5e0700
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:0849 MTU > 0
ib1: PathRec LID 0x0005 for GID fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Created ah 00000101b38a9f40
ib1: created address handle 00000101b2866f00 for LID 0x0005, SL 0
ib1: Request connection 0x8c05bd for gid fe80:0000:0000:0000:0005:ad00:0020:0849 qpn 0x404
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:0849
ib1: Destroy active connection 0x8c05bd head 0x0 tail 0x0
ib0: Created ah 00000101bdfbb9c0
ib0: Created ah 00000101b8b2e240
ib0: Created ah 00000101b2e9a340
ib0: Created ah 00000101b8b2e700
ib0: Created ah 00000101aa5e0140
ib0: enabling connected mode will cause multicast packet drops
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101b3983340
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101a3816880
ib0: Created ah 0000010037dbdb80
ib0: Created ah 00000101ab19f940
ib0: Created ah 00000101b3983500
ib0: Created ah 0000010037dbd9c0
ib0: enabling connected mode will cause multicast packet drops
ib1: Created ah 00000101b2ad7200
ib1: Created ah 00000101b2ad7680
ib1: Created ah 00000101be579c80
ib1: Created ah 00000101b2ad7180
ib1: Created ah 00000101b2e9aec0
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101bfc50180
ib1: Created ah 00000101be5d3240
ib1: Created ah 00000101b2543f40
ib1: Created ah 00000101be5d31c0
ib1: Created ah 00000101bf442280
ib1: enabling connected mode will cause multicast packet drops
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101bfc85600
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101ba22c4c0
ib1: Created ah 00000101b3983a00
ib1: Created ah 00000101b38a9f80
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Created ah 00000101a3816880
ib1: Created ah 00000101aa5e0280
ib1: Created ah 00000101b25433c0
ib1: Created ah 00000101b3983500
ib1: Created ah 00000101b39835c0
ib1: enabling connected mode will cause multicast packet drops
ib0: Created ah 00000101a3816c00
ib0: Created ah 00000101ab19f940
ib0: Created ah 00000101b3983a00
ib0: Created ah 00000101b2543980
ib0: Created ah 00000101bfc85600
ib1: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib1: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib1: Created ah 00000101bfc812c0
ib1: created address handle 00000101a7d47540 for LID 0x0007, SL 0
ib1: Request connection 0x8d05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib1: Reap connection for gid fe80:0000:0000:0000:0005:ad00:0020:084a
ib1: Destroy active connection 0x8d05bd head 0x0 tail 0x0
ib0: Created ah 00000101b2543f40
ib0: Created ah 00000101be5d3240
ib0: Created ah 00000101be5d3340
ib0: Created ah 00000101a3816b40
ib0: Created ah 00000101a3816f00
ib0: enabling connected mode will cause multicast packet drops
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101be977540
ib0: Created ah 00000101a3816580
ib0: Created ah 00000101ba22c4c0
ib0: Created ah 00000101a7d47540
ib0: Created ah 00000101be4d2a40
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101a3816e80
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101a3816680
ib0: Created ah 00000101aa5e0280
ib0: Created ah 00000101b2543980
ib0: Created ah 00000101b3983a00
ib0: Created ah 00000101ab19f940
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Port state change event
ib1: Port state change event
ib0: flushing
ib0: downing ib_dev
ib1: flushing
ib1: downing ib_dev
ib0: Created ah 00000101bfc85600
ib0: Created ah 0000010037dbd9c0
ib0: Created ah 00000101be579c80
ib0: Created ah 00000101a3816d00
ib0: Created ah 00000101a3816700
ib0: enabling connected mode will cause multicast packet drops
ib1: Created ah 00000101b2ad7700
ib1: Created ah 00000101b301ef80
ib1: Created ah 00000101a3816680
ib1: Created ah 00000101a3816c00
ib1: Created ah 00000101b2ad7200
ib0: Start path record lookup for fe80:0000:0000:0000:0005:ad00:0020:084a MTU > 0
ib0: PathRec LID 0x0007 for GID fe80:0000:0000:0000:0005:ad00:0020:084a
ib0: Created ah 00000101be5d3340
ib0: created address handle 00000101b2ad7680 for LID 0x0007, SL 0
ib0: Request connection 0x8e05bd for gid fe80:0000:0000:0000:0005:ad00:0020:084a qpn 0x405
ib0: CM error 0.
ib0: Destroy active connection 0x8e05bd head 0x0 tail 0x0

-- 
MST