[ofa-general] RE: [ofw] ib_create_qp and ib_get_err_str weirdness

[Tzachi Dar]

Hi Diego,

It seems that if you know the hw that you are working with you can find
(by experiments) the maximum number of sge that you can use. (probably
around 29).
So, you can limit your work requests to this number of SGE.

Depending if you are in user or in kernel you can also use buffers that
have the same Contiguous memory. (I don't know the control that you have
on the buffers so this is just a suggestion).

Thanks
Tzachi

> -----Original Message-----
> From: ofw-bounces at lists.openfabrics.org 
> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Diego Guella
> Sent: Friday, February 13, 2009 5:32 PM
> To: ofw at lists.openfabrics.org; OpenIB General
> Subject: [ofw] ib_create_qp and ib_get_err_str weirdness
> 
> Hello,
> 
> I am using Mellanox WinOF 2.0.0 with a MHES14-XTC SDR 
> single-port card.
> I noticed a strange behavior of ib_create_qp function:
> 
> -----
> memset(&qp_create, 0, sizeof(qp_create)); qp_create.qp_type = 
> IB_QPT_RELIABLE_CONN; // Reliable Connected 
> qp_create.sq_depth = ctx->qdepth; qp_create.rq_depth = 
> ctx->qdepth; qp_create.sq_sge = ctx->hca_attr->max_sges; 
> qp_create.rq_sge = ctx->hca_attr->max_sges; qp_create.h_sq_cq 
> = ctx->cq_h; qp_create.h_rq_cq = ctx->cq_h; qp_create.h_srq = 
> NULL; qp_create.sq_signaled = 1;
> ctx->qp_h = 0;
> rc = ib_create_qp(ctx->pd_h, &qp_create, NULL, NULL, &ctx->qp_h);
> -----
> return value ("rc") is 3 (=IB_INVALID_PARAMETER).
> 
> I spent some time figuring out the problem was the SQ SGE value:
> http://lists.openfabrics.org/pipermail/general/2006-June/023417.html
> 
> According to iba/ib_al.h:
> -----
> * IB_INVALID_MAX_SGE
> * The requested maximum number of scatter-gather entries for 
> the send or
> * receive queue could not be supported.
> -----
> so, why the return value isn't 22 (=IB_INVALID_MAX_SGE)?
> 
> In the discussion I mentioned, it turned out that even using 
> hca_attr->max_sges there is the possibility that ib_create_qp fails.
> Which is my case.
> I have the need to send some audio buffers (32 or more) from 
> an IO node to a computing node using RDMA WRITE.
> The ownership of the buffers is of the audio driver, and I 
> haven't the guarantee that the audio buffers are contiguous.
> I was trying and send them using the lowest possible number 
> of WR, each one with the highest possible number of sge.
> But, given the hca_attr->max_sge unreliability, how do you 
> recommend to achieve this goal?
> Should I post a WR for each buffer I'd want to send through 
> RDMA WRITE?
> 
> 
> Another less-related problem:
> ib_get_err_str is not correct for every input value, for 
> example I noticed that for
> ib_get_err_str(IB_INVALID_PD_HANDLE) the string returned is 
> IB_INVALID_MR_HANDLE
> 
> 
> I don't know if these problems apply to linux too, so I'm 
> including general list.
> 
> Thanks and best regards,
> Diego
> 
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>