[openib-general] RDMA RC QP returning "RNR Retry Counter Exceeded Error"

[Ira Weiny]

I have started writing a simple RDMA app which uses the rdmacm.  I have gotten
the connection established, QP's and MR's set up, and have sent the RDMA ETH.
However, more and more I am getting the RNR Retry Counter Exceeded error back
from the "client's" post send of the RDMA ETH.  About 1/10 times it will work
but most of the time it does not.  I have figured out that you can't set the
IBV_QP_RNR_RETRY attribute unless you go from RTR to RTS.  The state of the QP
is RTS and the IBV_QP_RNR_RETRY value is 0 as set by the rdmacm.  Do I have to,
or can I, transition the QP from RTS to RTR and then back again to set the
IBV_QP_RNR_RETRY?

Thanks,
Ira Weiny
weiny2 at llnl.gov