[openib-general] Re: Re: libibverbs and max inline data

[Michael S. Tsirkin]

Quoting r. James Lentini <jlentini at netapp.com>:
> Subject: Re: Re: libibverbs and max inline data
> 
> 
> 
> On Mon, 23 May 2005, Michael S. Tsirkin wrote:
> 
> >Quoting r. James Lentini <jlentini at netapp.com>:
> >>Subject: Re: Re: libibverbs and max inline data
> >>
> >>
> >>On Sun, 22 May 2005, Roland Dreier wrote:
> >>
> >>>  Michael> Hi!  I tried to figure out how much inline data a qp can
> >>>  Michael> support, to know what max_inline_data value to put in the
> >>>  Michael> qp.  However, there doesnt seem to exist a way to do
> >>>  Michael> that, currently.
> >>>
> >>>I think you're right.
> >>>
> >>>  Michael> Maybe the driver should be changed to trim
> >>>  Michael> max_inline_data to the maximum possible value?
> >>>
> >>>Yes, that would probably make sense.  In a sense that's what the
> >>>kernel driver already does -- it always clips the max_inline_data
> >>>request from the consumer down to 0.
> >>
> >>ib[v]_create_qp should return an error when an unsupported inline data
> >>size is requested.
> >
> >Why is it important? The user could always check the max_inline_data
> >returned and close the qp if thats too small, but I dont see
> >many users doing this - inline data is mainly a tuning opportunity.
> 
> Once there is a way to programmatically determine the max_inline_data 
> value, your proposal would be feasible.

There is a way to do it - the actual values are returned in the attr that you
pass to create_qp.

> Even if it were possible, I 
> still believe that the driver should validate this value.
> 
> Why have the user specify the ib[v]_qp_cap values if they are going to 
> be ignored?

max_inline_data is special in that its a tuning opportunity.
So its mostly harmless for the driver to ignore.

> As a rule, the driver should return an error when the user requests a 
> capability/resource that the hardware cannot support.
>
> kDAPL was bitten by this last week. The call to ib_create_qp succeed 
> but the value kDAPL was placing in the ib_qp_init_attr's 
> max_inline_data field was too large.

It was just a bug in the mthca kernel driver.
You shouldnt work around such bugs in an application.

> As a result, a subsequent qp call failed.
> If ib_create_qp had returned an error to begin with, it would 
> have been easier to determine the problem.

It seems wrong for the application to hard-code the max_inline_data
value that suits a particular hardware. Better for the app to tell
the driver what it needs and the driver will give it what hardware
can support.

> >>>Independent of that we could add a field to the device properties 
> >>>struct (and implement ibv_query_device() for userspace).

ibv_query_device would be useful for other things as well (e.g.
getting the max MTU).

-- 
MST - Michael S. Tsirkin