[ofa-general] [Bug report / partial patch] OFED 1.3 send max_sge lower than reported by ib_query_device

[Roland Dreier]

 > 1) ib_create_qp() fails with max_sge 
 >   If you use ib_query_device() to return the device specific 
 > attribute max_sge, it seems reasonable to expect you can create
 > a QP with max_send_sge=max_sge.  The problem is that this often
 > fails.
 > 
 >   The reason is that depending on the QP type (RC, UD, etc.) and
 > how the QP will be used (send, RDMA, atomic, etc.), there can be
 > extra segments required in the WQE that eat up SGE entries.  So
 > while some send WQE might have max_sge available SGEs, many will
 > not.

 >   This issue may need API extensions to definitively resolve.  In
 > the short term, it would be very nice if max_sge reported by 
 > ib_query_device() could always return a value that ib_create_qp()
 > could use.  Think of it as the minimum max_send_sge value that
 > will work for all QP types.

The intention is that any attempt to create a QP with the maximum
number of S/G entries as reported by query device should succeed.
However, as you note there may be issues that make this fail, but I
would consider them as bugs to be fixed.

You mention API extensions to handle this -- do you have any concrete
ideas?  In the past we've talked a little about this, but I don't
think anyone has suggested any changes that would help matters while
still keeping the API no more complex than it already is.

 >   The recent patch to support shrinking WQEs introduces a 
 > behavior that creates a big difference between the mlx4 
 > supported send SGEs (checked against 61, should be 59 or 60,
 > and reported in ib_query_device as 32 to equal receive side
 > max_rq_sg value).  

I'm not sure I understand this.  What's the new behavior?

Are you trying to take advantage of the fact that using non-power-of-2
size send WQEs would let you have a send queue with more than 32 S/G
entries?  I think doing that actually would require a change in the
API to allow different values for max_sge_rq and max_sge_sq to be
reported from ib_query_device().  Which in turn would break the
userspace ABI, etc, etc. and leaves me wondering if it's really worth it.

(BTW I hate the "shrinking WQE" terminology for this, although
obviously you weren't the one to introduce it)

 - R.