[openib-general] max_send_sge < max_sge

[Pete Wyckoff]

mst at mellanox.co.il wrote on Wed, 28 Jun 2006 17:51 +0300:
> Yea, that's because the API only can report 1 max value. But when
> this was considered the concensus was its not worth extending the API
> because of the other issues you mention.

Maybe you should report min(max_recv_sge, max_send_sge) instead of
max().  In this case I don't care because I currently need fewer
SGEs than either limit.  I'm just worried you're going to get the
same complaint by newbie IB users later.

> Yep.  We could have an option to have the stack scale the requested values down
> to some legal set instead of failing an allocation.  But we couldn't come up
> with a clean way to tell the stack e.g.  what should it round down: the SGE or
> WR value.  Do you think selecting something arbitrarily might still be a good
> idea?

No.  If I get fewer WRs than requested, the app would break.  If I
get fewer SGs, things would work for this particular app with some
more infrastructure to check for that, but I don't see how that
could be a general rule.

I like the model where the app can provision itself by querying the
NIC before opening any QPs, then get the same settings for every QP,
until the maximum number of QPs is reached.

We already have a way in PVFS2 to close "idle" connections, but it
isn't hooked up into QP allocation failure yet.  I prefer to do that
than to limp along on certain connections with fewer WRs or SGs,
along with all the code that would have to be added to handle that
situation.

> So in the end we are back to either using low numbers that just work
> empirically, or starting with some value and going down till it succeeds.

Yep.  Thanks for the insight.  It'll be fun when I try to get this
to work on amso with only 4 SGEs per QP.

		-- Pete