[openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support

[Michael S. Tsirkin]

Quoting r. Rimmer, Todd <trimmer at infiniconsys.com>:
> Subject: RE: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support
> 
> > Here's an updated patch - I'll be offline till Sunday and I hope
> > it's good to go. Please note there's a companion patch for libmthca,
> > ([PATCH 2/2]) which is also needed to actually use this stuff.
> > 
> > Quoting r. Roland Dreier <roland at topspin.com>:
> > > Subject: Re: [openib-general] [PATCH 1/2] mthca: 
> > max_inline_data support
> > > 
> > > Thanks... does it make sense to set the max_inline_data value for
> > > kernel QPs but then not support posting inline data?
> > 
> > You are right: max_inline_data in init_attr shall be left 0
> > until posting inline data in kernel is supported. Fixed.
> > 
> > > Also should we add some error checking against the HCA's maximum WQE
> > > size?
> > 
> > I added such a check.
> > 
> > 
> > Support max_inline_data parameter for userspace QPs.
> > Check max_send_sge/max_recv_sge/max_inline_data against the HCA's
> > limits.
> > 
> 
> Regarding inline data in posted wqes, this decision can be automatically made for user QPs and for some kernel QPs.  Hence the API need not be exposed in user space.
> 
> If all MRs in the Protection domain have CPU Virtual == IO Virtual, then the post_wqe routine can automatically inline the data as appropriate.  Given this situation, the IO Virtual address in the work request scatter gather list can be used as a direct CPU address to copy data into the WQE.
> 
> For user space QPs, CPU Virtual should always == IO Virtual.  This is because:
> register mr and register shared mr should use the CPU virtual as the IO Virtual.
> 
> In the kernel, a flag can be kept per PD indicating if CPU Virtual == IO Virtual.  Only for PDs using physical memory regions, would this condition be false.
> 
> This approach relieves applications from dealing with the inline decisions, and it can permit the verbs driver to inline data in some additional cases (such as when there are fewer scatter gather entries in a WQE, hence making more room for inline data).
> 
> Todd Rimmer
> 

There are 3 reasons this is not done this way, in order of relevance:
1. posting inline data is trading off higher CPU utilixation for lower
   latency, may be bad for some applications
2. inline data need not be registered, application can make use of this
   fact
3. driver would need to know whether the data will fit inline ahead
   of the time, which means an extra scan of the scatter/gather list,
   application typically knows the total size since it builds the list

-- 
MST - Michael S. Tsirkin