[ofa-general] Re: [ewg] rdma retry number

[Or Gerlitz]

Sunkyoung Shin wrote:
> During failover test, we found the iscsi over iser reconnected to the
> iscs target after 100 seconds due to the default max timeout (8sec) and
> retry number (15). The max timeout was adjustable with the module
> parameter, max_timeout, of ib_cm.ko, but the retry number wasn't. Can we
> add the retry number as module parameter of rdma_cm.ko? I added the
> patch below based on the ofed version, OFED-1.2-20070626-0917.

I understand that you want the QP timeout/retries to be smaller, and not 
the CM timeout/retries and hence there might be some confusion here 
which the following rdma-cm code snip from cma_connect_ib() might help 
resolving:
...
> 	req.qp_num = id_priv->qp_num;
>         req.qp_type = IB_QPT_RC;
>         req.starting_psn = id_priv->seq_num;
>         req.responder_resources = conn_param->responder_resources;
>         req.initiator_depth = conn_param->initiator_depth;
>         req.flow_control = conn_param->flow_control;
>         req.retry_count = conn_param->retry_count;
>         req.rnr_retry_count = conn_param->rnr_retry_count;
>         req.remote_cm_response_timeout = CMA_CM_RESPONSE_TIMEOUT;
>         req.local_cm_response_timeout = CMA_CM_RESPONSE_TIMEOUT;
>         req.max_cm_retries = CMA_MAX_CM_RETRIES;
>         req.srq = id_priv->srq ? 1 : 0;
> 
>         ret = ib_send_cm_req(id_priv->cm_id.ib, &req);
...

The user is in total control on the QP retry count through the rdma-cm 
connection param structure, the req.max_cm_retries has nothing to do 
with the QP timeout.

The RC QP timeout is derived by the IB CM internally (on ofed through 
module param which you have changed) and the rdma-cm nor its consumer 
have direct control on it.

This follows the IB spec spirit that the SM/SA is the one to calculate 
and return to the host a param named "this path packet life time" so the 
IB CM combines the packet life time and something called the "hca ack 
delay". Currently the IB CM just 2 * path.packet_life_time as an 
estimation for the timeout which is the packet life time plus the hca 
ack delay, see cm_init_av_by_path() in core/cm.c .

Note that the actual timeout T = 4.096us * 2^t where t is the value 
plugged into the QP. Hence doing t = path.packet_life_time + 1 does what 
I described above.

In examination I did on the past I think that the openSM always returns
path.packet_life_time = 18 and same for some vendor SMs. This means that 
the timeout is 2^(2+18+1) = 2^21us = 2 seconds

The # retries set by the iser initiator are seven (see 
iser_route_handler()) so seven times two give 14 seconds, which makes 
your report on the 100 seconds it took the initiator to reconnect to 
possibly point on the different problem.

Or.