[ofa-general] Re: Incorrect max_sge reported in mthca device query

[Michael S. Tsirkin]

> Quoting Tom Tucker <tom at opengridcomputing.com>:
> Subject: Incorrect max_sge reported in mthca device query
> 
> 
> Roland:
> 
> I think the max_sge reported by mthca_query_device is off by one. If you
> try to create a QP with the reported max, it fails with -EINVAL. I think
> the reason is that the mthca_alloc_wqe_buf function reserves a slot for
> a "bind request" and this pushes the WQE size over the 496B limit when
> the user requests the max (30) when allocating the QP.
> 
> Please let me know if I'm confused about what max_sge really means.
> 
> Thanks,
> Tom

Tom,
	max_sge reported by mthca_query_device is the upper bound
	for all QP types. I have not tested this, but think you can
       	create a UD type QP with this number of SGEs.

	I'd like to add that there can be no hard guarantee that
	creating a QP with a specific set of max_sge/max_wr always
	succeeds even if it is within the range of values reported
	by mthca_query_device: for example, for userspace QPs, the
       	system administrator might have limited the amount of
	memory that can be locked up by these QPs, and
	QP allocation requests with large max_sge/max_wr
	values will always fail. There are other examples of this.
	Thus, an application that wants to use as large a number of SGEs/WRs as
	possible in a robust fashion currently has no other choice except
       	a trial and error approach, handling failures gracefully.

	Finally, as a side note, it is *also* inefficient to request
	allocation of more sge entries than ULP will typically
	use - for reasons such as cache utilization, and many others.
       	How does this overhead trade-off against the need to sometimes post
	multiple WRs by ULP will depend both on ULP and the hardware
	used. This need to tune the ULP to a specific HCA is annoying,
	and might be something that we want to try and solve at
	the API level. However, max_sge/max_wr values in query device
	are unlikely to be the appropriate API for this.

	One way out could be to extend the API for create_qp and friends,
	passing in both min and max values for some parameters,
	and allowing the verbs provider to choose the optimal combination
	of these. I think I floated a similiar proposal once already, but there
	didn't appear to be sufficient user support for such a large API
	extension.

-- 
MST