[openib-general] Re: Re: libibverbs and max inline data

[James Lentini]

On Mon, 23 May 2005, Michael S. Tsirkin wrote:

> Quoting r. James Lentini <jlentini at netapp.com>:
>> Subject: Re: Re: libibverbs and max inline data
>>
>>
>>
>> On Mon, 23 May 2005, Michael S. Tsirkin wrote:
>>
>>> Quoting r. James Lentini <jlentini at netapp.com>:
>>>> Subject: Re: Re: libibverbs and max inline data
>>>>
>>>>
>>>> On Sun, 22 May 2005, Roland Dreier wrote:
>>>>
>>>>>  Michael> Hi!  I tried to figure out how much inline data a qp can
>>>>>  Michael> support, to know what max_inline_data value to put in the
>>>>>  Michael> qp.  However, there doesnt seem to exist a way to do
>>>>>  Michael> that, currently.
>>>>>
>>>>> I think you're right.
>>>>>
>>>>>  Michael> Maybe the driver should be changed to trim
>>>>>  Michael> max_inline_data to the maximum possible value?
>>>>>
>>>>> Yes, that would probably make sense.  In a sense that's what the
>>>>> kernel driver already does -- it always clips the max_inline_data
>>>>> request from the consumer down to 0.
>>>>
>>>> ib[v]_create_qp should return an error when an unsupported inline data
>>>> size is requested.
>>>
>>> Why is it important? The user could always check the max_inline_data
>>> returned and close the qp if thats too small, but I dont see
>>> many users doing this - inline data is mainly a tuning opportunity.
>>
>> Once there is a way to programmatically determine the max_inline_data
>> value, your proposal would be feasible.
>
> There is a way to do it - the actual values are returned in the attr that you
> pass to create_qp.

See my previous email to Roland. I believe that mthca_create_qp does 
not update the attr if an error occurs.

>
>> Even if it were possible, I
>> still believe that the driver should validate this value.
>>
>> Why have the user specify the ib[v]_qp_cap values if they are going to
>> be ignored?
>
> max_inline_data is special in that its a tuning opportunity. So its 
> mostly harmless for the driver to ignore.

The driver doesn't ignore it though. Take a look at mthca_set_qp_size. 
If you set max_inline_data to a value such that qp->sq.max_gs will be 
dev->limits.max_sg, then mthca_set_qp_size will fail with error 
EINVAL. This will result in the failure of ib_create_qp.

>> As a rule, the driver should return an error when the user requests a
>> capability/resource that the hardware cannot support.
>>
>> kDAPL was bitten by this last week. The call to ib_create_qp succeed
>> but the value kDAPL was placing in the ib_qp_init_attr's
>> max_inline_data field was too large.
>
> It was just a bug in the mthca kernel driver.
> You shouldnt work around such bugs in an application.

I'm sorry. My memory was wrong here. ib_create_qp did fail when the 
max_inline_data field was too large.

>
>> As a result, a subsequent qp call failed.
>> If ib_create_qp had returned an error to begin with, it would
>> have been easier to determine the problem.
>
> It seems wrong for the application to hard-code the max_inline_data
> value that suits a particular hardware. Better for the app to tell
> the driver what it needs and the driver will give it what hardware
> can support.
>
>>>>> Independent of that we could add a field to the device properties
>>>>> struct (and implement ibv_query_device() for userspace).
>
> ibv_query_device would be useful for other things as well (e.g.
> getting the max MTU).
>
> -- 
> MST - Michael S. Tsirkin
>