[ofa-general] proper way to recover from poll CQ failed error

Dotan Barak dotanb at dev.mellanox.co.il
Thu Mar 13 07:02:26 PDT 2008


Can you execute ibv_devinfo after you get this error?
(I'm trying to reproduce this error in our labs without any success so far).

can you please send me the output of "uname -a" + "cat /proc/cpuinfo"


Dotan



Murray Smigel wrote:
> Dotan Barak wrote:
>
>> Hi.
>>
>> The fact that ibv_poll_cq failed indicates that something bad happened.
>> Usually this failure should create any problem and only the process 
>> that had the problem is being
>> effected from this.
>>
>> I personally think that the ib_* performance tools are better to 
>> check the performance of your subnet.
>
> **
> Which programs are these?
>
>>
>> I will be happy if you'll answer the following questions:
>> Is this error is consistent?
>
> **
> Yes. From this morning...
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 1 -n 1000
> ...
> 2000 bytes in 0.00 seconds = 3.56 Mbit/sec
> 1000 iters in 0.00 seconds = 4.49 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 100 -n 1000
> ...
> 200000 bytes in 0.00 seconds = 357.46 Mbit/sec
> 1000 iters in 0.00 seconds = 4.48 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 200 -n 1000
> ...
> 400000 bytes in 0.00 seconds = 711.74 Mbit/sec
> 1000 iters in 0.00 seconds = 4.50 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 500 -n 1000
> ...
> 1000000 bytes in 0.00 seconds = 1752.08 Mbit/sec
> 1000 iters in 0.00 seconds = 4.57 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 1000 -n 1000
> ...
> 2000000 bytes in 0.01 seconds = 2973.98 Mbit/sec
> 1000 iters in 0.01 seconds = 5.38 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 2000 -n 1000
> ...
> 4000000 bytes in 0.01 seconds = 5226.20 Mbit/sec
> 1000 iters in 0.01 seconds = 6.12 usec/iter
>
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 5000 -n 1000
> ...
> poll CQ failed -2
>
> Now, this used to work:
> murray at nasnu2:/usr/local/bin$ ./ibv_srq_pingpong -s 1 -n 1000
> ...
> poll CQ failed -2
>
> As did this:
> murray at nasnu2:/usr/local/bin$ ./ibv_rc_pingpong -s 1 -n 1000
>  local address:  LID 0x0003, QPN 0xa9004a, PSN 0x540960
>  remote address: LID 0x0004, QPN 0xaf004a, PSN 0x787c46
> poll CQ failed -2
>
>> Can you please send me the output of the ibv_devinfo of your machines?
>
> murray at nasnu2:/usr/local/bin$ ./ibv_devinfo
> hca_id: mlx4_0
>        fw_ver:                         2.3.000
>        node_guid:                      0002:c903:0000:c51c
>        sys_image_guid:                 0002:c903:0000:c51f
>        vendor_id:                      0x02c9
>        vendor_part_id:                 25418
>        hw_ver:                         0xA0
>        board_id:                       MT_04A0120002
>        phys_port_cnt:                  2
>                port:   1
>                        state:                  PORT_ACTIVE (4)
>                        max_mtu:                2048 (4)
>                        active_mtu:             2048 (4)
>                        sm_lid:                 3
>                        port_lid:               3
>                        port_lmc:               0x00
>
>                port:   2
>                        state:                  PORT_DOWN (1)
>                        max_mtu:                2048 (4)
>                        active_mtu:             2048 (4)
>                        sm_lid:                 0
>                        port_lid:               0
>                        port_lmc:               0x00
>
>
> murray at nasnu3:~$ cd /usr/local/bin
> murray at nasnu3:/usr/local/bin$ ./ibv_devinfo
> hca_id: mlx4_0
>        fw_ver:                         2.3.000
>        node_guid:                      0002:c903:0000:c474
>        sys_image_guid:                 0002:c903:0000:c477
>        vendor_id:                      0x02c9
>        vendor_part_id:                 25418
>        hw_ver:                         0xA0
>        board_id:                       MT_04A0120002
>        phys_port_cnt:                  2
>                port:   1
>                        state:                  PORT_ACTIVE (4)
>                        max_mtu:                2048 (4)
>                        active_mtu:             2048 (4)
>                        sm_lid:                 3
>                        port_lid:               4
>                        port_lmc:               0x00
>
>                port:   2
>                        state:                  PORT_DOWN (1)
>                        max_mtu:                2048 (4)
>                        active_mtu:             2048 (4)
>                        sm_lid:                 0
>                        port_lid:               0
>                        port_lmc:               0x00
>
>
>> Did you have any error message in the /var/log/messages when you saw 
>> this error?
>
>
>
> nasnu2:/usr/local/bin# tail /var/log/messages
> Mar 13 05:30:05 nasnu2 -- MARK --
> Mar 13 05:50:05 nasnu2 -- MARK --
> Mar 13 06:10:05 nasnu2 -- MARK --
> Mar 13 06:30:05 nasnu2 -- MARK --
> Mar 13 06:50:05 nasnu2 -- MARK --
> Mar 13 07:10:05 nasnu2 -- MARK --
> Mar 13 07:30:05 nasnu2 -- MARK --
> Mar 13 07:36:09 nasnu2 syslogd 1.4.1#18: restart.
> Mar 13 07:50:05 nasnu2 -- MARK --
> Mar 13 08:10:05 nasnu2 -- MARK --
>
> nasnu2:/usr/local/bin# tail /var/log/opensm.log
> Mar 13 08:16:46 024587 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:16:56 024615 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:06 024596 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:16 024549 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:26 024664 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:36 024626 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:46 024627 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:17:56 024611 [43806960] 0x02 -> SUBNET UP
> Mar 13 08:18:06 024607 [43806960] 0x02 -> SUBNET UP
>
> Thanks for your help,
> murray smigel
>
>
>>
>> thanks
>> Dotan
>>
>> Murray Smigel wrote:
>>
>>> Hi,
>>> I am running OFED-3.0 using ConnectX adapters in a two machine 
>>> direct connect mode.
>>> Most of the various pingpong tests seem ok, but when I run
>>> ibv_srq_pingpong -s 500 -n 1000
>>>
>>> I get poll "CQ failed -2" when I start up the client side.  Smaller 
>>> values of -s worked fine.
>>> Once this happens, no other pingpong tests seem to work.
>>> I have then unloaded all the ib_* mlmx_* and iw_* modules, reloaded 
>>> them and things still
>>> fail. I have to reboot the machines to get things back.
>>>
>>> 1) Is there a cleaner way to recover from this situation?
>>> 2) Is the initial failure an indication that something else is wrong?
>>> 3) Is the -s 1 latency I see with ibv_rc_pingpong of ~7 microseconds 
>>> reasonable?
>>>
>>> Thanks,
>>> murray smigel
>>>
>>>
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit 
>>> http://openib.org/mailman/listinfo/openib-general
>>>
>>
>
>




More information about the general mailing list