[ofa-general] ib_create_qp and ib_get_err_str weirdness

[Dotan Barak]

Hi.

Diego Guella wrote:
> Hello,
>
> I am using Mellanox WinOF 2.0.0 with a MHES14-XTC SDR single-port card.
> I noticed a strange behavior of ib_create_qp function:
>
> -----
> memset(&qp_create, 0, sizeof(qp_create));
> qp_create.qp_type = IB_QPT_RELIABLE_CONN; // Reliable Connected
> qp_create.sq_depth = ctx->qdepth;
> qp_create.rq_depth = ctx->qdepth;
> qp_create.sq_sge = ctx->hca_attr->max_sges;
> qp_create.rq_sge = ctx->hca_attr->max_sges;
> qp_create.h_sq_cq = ctx->cq_h;
> qp_create.h_rq_cq = ctx->cq_h;
> qp_create.h_srq = NULL;
> qp_create.sq_signaled = 1;
> ctx->qp_h = 0;
> rc = ib_create_qp(ctx->pd_h, &qp_create, NULL, NULL, &ctx->qp_h);
> -----
> return value ("rc") is 3 (=IB_INVALID_PARAMETER).
>
> I spent some time figuring out the problem was the SQ SGE value:
> http://lists.openfabrics.org/pipermail/general/2006-June/023417.html
>
> According to iba/ib_al.h:
> -----
> * IB_INVALID_MAX_SGE
> * The requested maximum number of scatter-gather entries for the send or
> * receive queue could not be supported.
> -----
> so, why the return value isn't 22 (=IB_INVALID_MAX_SGE)?
>
> In the discussion I mentioned, it turned out that even using 
> hca_attr->max_sges there is the possibility that ib_create_qp fails.
> Which is my case.
> I have the need to send some audio buffers (32 or more) from an IO 
> node to a computing node using RDMA WRITE.
> The ownership of the buffers is of the audio driver, and I haven't the 
> guarantee that the audio buffers are contiguous.
> I was trying and send them using the lowest possible number of WR, 
> each one with the highest possible number of sge.
> But, given the hca_attr->max_sge unreliability, how do you recommend 
> to achieve this goal?
I saw code that is aware to this problem and try to create a QP with the 
maximum number of sge, and upon failures, decrease this value
until the QP can be created.

If you will use maximum supported number of sge minus a constant (let's 
say: 2), it should be always o.k..

> Should I post a WR for each buffer I'd want to send through RDMA WRITE?
If you are talking about local buffers, than you can use send data from 
several buffers using the same SR.
If you are talking about remote buffers, than you have to use different 
SR for every remote buffer that you want to fill.
>
>
> Another less-related problem:
> ib_get_err_str is not correct for every input value, for example I 
> noticed that for
> ib_get_err_str(IB_INVALID_PD_HANDLE) the string returned is 
> IB_INVALID_MR_HANDLE
>
>
> I don't know if these problems apply to linux too, so I'm including 
> general list.
In Linux the return values are different (usually, -1 means that there 
is an error and that's all...).
I believe that the error exists only in the win-ofa code.

Dotan