[ofa-general] mthca max_sge value... ugh.

[Jack Morgenstein]

This is actually a known issue, which we never got around to fixing.

From the OFED 1.2.5 release notes (document docs/mthca_release_notes.txt),
in section "3. Known Issues":

3. In mem-free devices, RC QPs can be created with a maximum of (max_sge - 3)
   entries only.

- Jack

On Friday 16 May 2008 10:19, Olaf Kirch wrote:
> On Friday 16 May 2008 00:22:12 Roland Dreier wrote:
> 
> I ran into this a few weeks back as well, when I tried to up the SG limit
> in RDS to 32 (on a arbel memfree card).

memfree returns 30 as the max, I believe.  ConnectX returns 32.
 
> I grepped around the code a bit, got a little confused because of all the
> different max_sge, max_sg and max_gs variables :-) and eventually
> convinced myself that the max_sge reported simply doesn't include the
> transport specific overhead that mthca_alloc_wqe_buf factors in.
> 
> Given that you have quite different WQE overheads depending on the transport,
> a conservative max_sge value that works for all transports wastes one or two
> entries on some others. Maybe once the QP is created, it could report
> the actual max_sge value (which may actually be greater than the conservative,
> transport-independent max_sge estimate of the device).

This is a problem, because then you are returning a value which is greater that the
declared device max.  This causes IB Spec non-compliance.

> 
> Olaf