[openib-general] max_send_sge < max_sge

[Pete Wyckoff]

mst at mellanox.co.il wrote on Wed, 28 Jun 2006 01:38 +0300:
> If this works for you, great. I was just trying to point out query device can
> not guarantee that QP allocaton will always succeed even if you stay within
> limits it reports.
> 
> For example, are you using a large number of WRs per QP as well?  If so after
> alocating a couple of QPs you might run out of locked memory limit allowed
> per-user, depending on your system setup. QP allocation will then fail, even if
> you use the hcacap - 1 heuristic.

Thanks for all the comments.  I'm not specifically trying to be a
pain here.  The bit I was failing to notice was that when
considering many QP allocations, the resource demands add up faster
when using more SGEs each.  Still find it odd that the very first
QP created can not achieve the maximum-reported values, but
understand your general argument.

Regarding the API, some interfaces I've seen will do the equivalent
of putting the "max currently available" values in ibv_qp_init_attr
so userspace can reconsider and try again.  I never liked that very
much, and it doesn't help much in this multi-dimensional space where
WRs and SGEs apparently share the same overall constraints.  Plus
the returned values aren't guaranteed to be valid next time an
attempt is made anyway, so don't do that.  :)

It may make people realize what's going on faster to get an actual
return value somewhere.  Right now many failure conditions are
lumped into the returned NULL pointer: attr->cap values are bigger
than HCA max, a library malloc falied, the HCA is out of new QP
resources, the HCA is on fire.  That said, an API that returns an
explicit error code is clumsy:

    int ibv_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr
	*qp_init_attr, struct ibv_qp **newqp);

    struct ibv_qp *qp;
    int ret = ibv_create_qp(pd, &attr, &qp);
    if (ret < 0)
	printf("create qp failed: %s", strerror(ret));

So I'll have to vote against that bad idea too.

It would be possible but odd to store the return code in errno.
I.e., use the current API, but augmented to stick the return value
in the (thread-private) errno.  I'm not sure if I've seen anything
outside of libc use errno.  Having "ibv_errno" would be icky.

Thanks,

		-- Pete