<div dir="ltr">Hi Guys,<div><br></div><div>I had a user report that his jobs failed overnight and saw the following error message in his logs:</div><div><br></div><div>--</div><div><div><font color="#cc0000" face="courier new, monospace" size="1">--------------------------------------------------------------------------<br>
The OpenFabrics stack has reported a network error event. Open MPI<br>will try to continue, but your job may end up failing.<br>
<br> Local host: amber04<br> MPI process PID: 23493<br>
Error number: 10 (IBV_EVENT_PORT_ERR)<br><br>This error may indicate connectivity problems within the fabric;<br>
please contact your system administrator.<br>--------------------------------------------------------------------------<br>
[amber04:23491] 5 more processes have sent help message help-mpi-btl-openib.txt / of error <br>event<br>
[amber04:23491] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error m<br>essages<br>
[[54699,1],1][btl_openib_component.c:3224:handle_wc] from amber03 to: amber04 error polling<br>
LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 16045312 opcode 0 vendo<br>r error 129 qp_idx 0<br>
--------------------------------------------------------------------------<br>
The InfiniBand retry count between two MPI processes has been<br>exceeded. "Retry count" is defined in the InfiniBand spec 1.2<br>
(section 12.7.38):<br><br> The total number of times that the sender wishes the receiver to<br>
retry timeout, packet sequence, etc. errors before posting a<br> completion error.<br>
<br>This error typically means that there is something awry within the<br>InfiniBand fabric itself. You should note the hosts on which this<br>
error has occurred; it has been observed that rebooting or removing a<br>particular host from the job can sometimes resolve this issue. <br>
<br>Two MCA parameters can be used to control Open MPI's behavior with<br>respect to the retry count:<br>
<br>* btl_openib_ib_retry_count - The number of times the sender will<br> attempt to retry (defaulted to 7, the maximum value).<br>
* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted<br> to 10). The actual timeout value used is calculated as:<br>
<br> 4.096 microseconds * (2^btl_openib_ib_timeout)<br><br>
See the InfiniBand spec 1.2 (section 12.7.34) for more details.<br>Below is some information about the host that raised the error and the<br>
peer to which it was connected:<br><br> Local host: amber03<br>
Local device: mlx4_0<br> Peer host: amber04</font></div></div><div>--<br></div><div><br></div><div style>When I checked the logs on our Mellanox IB Switch, I saw the following errors during the same time:</div><div style>
<br></div><div style>--</div><div style>Apr 8 22:26:53 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.WARNING]:
refresh_i2c(), hwd_main.c:7345, build 1: can't refresh device error
MLXI2C_CR_ERROR 8. closing and reseting device<br>
Apr 8 22:26:53 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.WARNING]:
iterate_temp_sensors(), hwd_main.c:11257, build 1: Failed to refresh i2c
device MLXI2C_CR_ERROR 8<br>Apr 8 22:26:53 ib01-oo74 hwd[2244]: TID
1208100608: [hwd.ERR]: iterate_temp_sensors(), hwd_main.c:11257, build
1: Error code 8 returned<br>
Apr 8 22:26:53 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.ERR]:
hwd_mon_handle_iterate(), hwd_main.c:10618, build 1: Error code 8
returned<br>Apr 8 22:26:53 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.ERR]: mdc_mon_iterate_node_internal(), mdc_misc.c:586, build 1: Error code 8 returned<br>
Apr 8 22:28:12 ib01-rwc-oo74 temp_control[2323]: [tc.ERR]:
get_bindings_by_name(), tc.c:36, build 1: Received empty data
system/chassis/temperature/state<br>Apr 8 22:28:12 ib01-oo74 temp_control[2323]: [tc.WARNING]: Failed to get binding: system/chassis/temperature/state err:0<br>
Apr 8 22:28:12 ib01-oo74 temp_control[2323]: [tc.ERR]:
lew_universal_event_handler(), libevent_wrapper.c:303, build 1: Error
code 6 returned<br>Apr 8 22:28:32 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.WARNING]: iterate_is4modules_initialized(), hwd_main.c:11368, build 1: Failed to refresh i2c device MLXI2C_ERROR 1<br>
Apr 8 22:28:32 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.ERR]: iterate_is4modules_initialized(), hwd_main.c:11368, build 1: Error code 1 returned<br>Apr
8 22:28:32 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.ERR]:
hwd_mon_handle_iterate(), hwd_main.c:10659, build 1: Error code 1
returned<br>
Apr 8 22:28:32 ib01-oo74 hwd[2244]: TID 1208100608: [hwd.ERR]: mdc_mon_iterate_node_internal(), mdc_misc.c:586, build 1: Error code 1 returned<br>Apr
8 22:29:45 ib01-oo74 smm[2275]: [smm.ERR]: smm_forward_sa_db(),
smm_main.c:7167, build 1: Error code 14004 (item not found) returned<br>
Apr 8 22:29:45 ib01-oo74 smm[2275]: [smm.ERR]: smm_get_sa_db(),
smm_main.c:6748, build 1: Error code 14004 (item not found) returned<br>Apr
8 22:29:45 ib01-oo74 smm[2275]: [smm.ERR]:
smm_handle_sa_db_updates(), smm_main.c:7253, build 1: Error code 14004
(item not found) returned<br>
Apr 8 22:29:45 ib01-oo74 smm[2275]: [smm.ERR]:
smm_handle_event_request(), smm_main.c:7393, build 1: Error code 14004
(item not found) returned <br></div><div style>--</div><div style><br></div><div style>Not sure what these error codes are and could not find anything on the web. Any ideas what these error messages are and if they would cause a disconnect on the ib network? I have run perfquery's and ibdiagnet and the only thing that I can see is some nodes exceeding the PortXmitDiscards. I have reset these counters to see if this is still happening but don't really think that is the cause of the problem here.</div>
<div style><br></div><div style>Any help would be greatly appreciated.</div><div style><br></div><div style>Thanks.</div></div>