[openib-general][kdapl]: vmalloc instead of kmalloc

[James Lentini]

On Sun, 7 Aug 2005, Guy German wrote:

> Hi,
>
> James Lentini <mailto:jlentini at netapp.com> wrote:
>> On Thu, 4 Aug 2005, Guy German wrote:
>>
>>> James,
>>>
>>> I see what you mean.
>>> The allocation of the event vector is derived from evd->qlen.
>>> In DTO ev'd, however, qlen is also the parameter passed to
>>> ib_create_cq. Since we don't want to limit DAPL consumers to an
>>> unnecessary small completion queue size, maybe we
>>> could differentiate between DTO supporting evd's and
>>> CONN evd's, when allocating the events vector.
>>>
>>> if evd supports CONN only, leave it :
>>> 	event = kmalloc(evd->qlen * sizeof *event)
>>> (Relying on the consumer he knows what he is doing)
>>> if evd is DTO only :
>>> 	don't allocate an event buffer, at all
>>> if evd supports both :
>>> 	event = kmalloc(DEFAULT_4_CONN * sizeof *event)
>>
>> And dynamically add additional events up to qlen as needed?
>>>
>>> if DEFAULT_4_CONN=256, that's a 3 pages allocation.
>>>
>>> How does that sound to you ?
>>
>> I'd prefer that the EVDs were uniform. I would worry about bugs
>> otherwise.
>
> But if it's the right thing to do (allocating less, or not 
> allocating at all), wouldn't it be best to try and fix those bugs ?

I think that the dynamic allocation strategy could be used for all 
types of EVDs.

EVDs that only handled DTO events (events generated by an IB CQ) would 
have an initial allocation of 0 DAT events. These EVDs would never 
grow the event pool.

EVDs that handled "software" events might grow this event pool. For 
these EVDs we could allocate some number of initial events.

The advantage of making the EVDs uniform is that the de-allocation 
code would be the same.

>> The eventual solution has to support qlen generated events (connection
>> request, connection, disconnect, software) if those event types are
>> supported by the EVD (even if the EVD is being used for both generated
>> events and DTOs).
>
> I think that there might be a mix up in the qlen meaning, for the consumer.
> ia_quey returns cq_len parameter, even if the evd in mind is used
> for connection only. In that case the result of qlen (128k), from the hca
> has nothing to do with the real resources availability.

I assume you are referring to the dat_ia_attr's max_evd_qlen value. You 
are right that it is not entirely accurate to report the CQ max qlen 
as this value. As you point out, there are really two limits: the 
maximum number of DTO and RMR events and the maximum number of 
"software" events. The former are limited by the capabilities of the 
underlying CQ while the later are constrained by the systems 
resources.

> You mentioned before that "allocating an event pool equal to
> the queue length seems like overkill", and I agree.
>
> If you want to support pending events list of more than ~2700, just 
> for CONN events, and not use vmalloc, we need some way of doing a 
> few kmallocs and managing that. Is it really necessary ? wouldn't 
> 256 *pending* events for CONN. purposes be enough ?

I don't know what the consumer requirements are this value. Remember 
that there are more "software" events than just pending connection 
events. All of these event types correspond to what I have been 
terming "software" events:

         DAT_CONNECTION_REQUEST_EVENT = 0x02001,
         DAT_CONNECTION_EVENT_ESTABLISHED = 0x04001,
         DAT_CONNECTION_EVENT_PEER_REJECTED = 0x04002,
         DAT_CONNECTION_EVENT_NON_PEER_REJECTED = 0x04003,
         DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR = 0x04004,
         DAT_CONNECTION_EVENT_DISCONNECTED = 0x04005,
         DAT_CONNECTION_EVENT_BROKEN = 0x04006,
         DAT_CONNECTION_EVENT_TIMED_OUT = 0x04007,
         DAT_CONNECTION_EVENT_UNREACHABLE = 0x04008,
         DAT_ASYNC_ERROR_EVD_OVERFLOW = 0x08001,
         DAT_ASYNC_ERROR_IA_CATASTROPHIC = 0x08002,
         DAT_ASYNC_ERROR_EP_BROKEN = 0x08003,
         DAT_ASYNC_ERROR_TIMED_OUT = 0x08004,
         DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR = 0x08005,
         DAT_SOFTWARE_EVENT = 0x10001

If place a limit (be it 256 or some other value) on the number of 
events of the types above, there will be a consumer that will want 
more.

My goal is that the queue length of an EVD supporting all events types 
(DTO and software) only be limited by the capabilities of the 
underlying CQ.