[ofa-general] Retry count error with ipath on OFED-1.3

[Keshetti Mahesh]

OFED 1.3 Infinipath Error:
 ># OSU MPI Bandwidth Test v3.1
 ># Size        Bandwidth (MB/s)
 >1                         0.17
 >2                         0.39
 >4                         0.66
 >8                         1.80
 >16                        2.53
 >32                        5.11
 >64                        8.80
 >128                      23.09
 >256                      43.65
 >512                      84.42
 >1024                    151.63
 >[0,1,0][btl_openib_component.c:1338:btl_openib_component_progress]
from >compute-6-7 to: compute-6-8 error polling HP CQ with status RETRY
 >EXCEEDED ERROR status number 12 for wr_id 185705200 opcode 1
 >--------------------------------------------------------------------------

 >The InfiniBand retry count between two MPI processes has been
 >exceeded.  "Retry count" is defined in the InfiniBand spec 1.2
 >(section 12.7.38):
 >
 >    The total number of times that the sender wishes the receiver to
 >    retry timeout, packet sequence, etc. errors before posting a
 >    completion error.
 >
 >This error typically means that there is something awry within the
 >InfiniBand fabric itself.  You should note the hosts on which this
 >error has occurred; it has been observed that rebooting or removing a
 >particular host from the job can sometimes resolve this issue.
 >
 >Two MCA parameters can be used to control Open MPI's behavior with
 >respect to the retry count:
 >
 >* btl_openib_ib_retry_count - The number of times the sender will
 >  attempt to retry (defaulted to 7, the maximum value).
 >
 >* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted
 >  to 10).  The actual timeout value used is calculated as:
 >
 >     4.096 microseconds * (2^btl_openib_ib_timeout)
 >
 >  See the InfiniBand spec 1.2 (section 12.7.34) for more details.
 >--------------------------------------------------------------------------

 >mpirun noticed that job rank 1 with PID 16883 on node compute-6-8
 >exited on signal 15 (Terminated).


Hi Nico,

The above error arises due to dead lock scenario in the network. But that
should not happen in your case since you are using only 2 nodes.
Try increasing the IB parameters (btl_openib_ib_retry_count and
btl_openib_ib_timeout) mentioned in the error.

-Mahesh