[openib-general][kdapl]: vmalloc instead of kmalloc

[James Lentini]

On Wed, 3 Aug 2005, Guy German wrote:

> James Lentini <mailto:jlentini at netapp.com> wrote:
>>
>> kDAPL creates two large pools of memory.
>>
>> One is for events. When the kDAPL consumer creates an EVD, it
>> specifies a queue size (the number of events the EVD can hold). The
>> implementation pre-allocates a pool of events equal to the
>> size of the
>> queue. These events are used when an IB upcall is made (e.g.
>> connection request, connection established, aysnc. error,
>> etc.) or the
>> kDAPL consumer posts a "software event" via dat_evd_post_se().
>
> And, of course  - completions of data events - which is why the queue
> need to be more substantial.

In uDAPL yes, but not in kDAPL. None of the callers of 
dapl_evd_get_event(), where events are dequeued from the EVD's 
free_event_queue, use the events for DTO completions.

In uDAPL, dat_evd_wait can dequeue data events and store them in the 
pending event queue.

As Itamar pointed out, kDAPL could use a single circular list instead 
of maintaining the EVD's free and pending event queues.

>> The other memory pool is for cookies. A kDAPL event contains certain
>> fields that the IB work completion (ib_wc) does not provide (like the
>> EVD, EP, etc.). For that reason, the kDAPL provider sticks
>> the missing
>> information in a dapl_cookie structure and sets it as the work
>> request's context value. When the work completion comes back, the
>> kDAPL provider pulls the cookie out and uses it to populate the
>> missing event fields. These cookies are also pre-allocated in a pool
>> equal to the EVD size.
>>
>>>> To answer your question, vmalloc has a performance overhead and can
>>>> and will fail when vmalloc-space is exhausted (as can kmalloc, for
>>>> different reasons). Can this allocation be cut down so that it
>>>> becomes a non-issue?
>>
>> The size of the event pool seems much larger than necessary. I would
>> expect most consumers only use a few events from this pool (with no
>> errors or software events, a client will use 2 and a server will use
>> 3).
>
> If you consider that the consumers are polling from the queue 
> themselves (upcall policy is disabled, for performance) and the 
> queue of events holds completions of data, then you have to support 
> larger queue. Bare in mind that one target can have many initiators.

Even if the event queue never stores DTO events? In the worst case, I 
agree that kDAPL would need to allocate an amount of memory equal to n 
* sizeof(DAT_EVENT), where n is the EVD queue size.

My observation is that an EVD almost always uses the event pool < 5 
times (when there are no async errors and no software events). Further 
more, it usually only uses one event at a time (it posts a connection 
request event, the consumer reaps it, it posts a connection event, the 
consumer reaps it, ...). Given that, allocating a event pool equal to 
the queue length seems like overkill to me. The EVD could allocate 
smaller blocks of events in some configurable size. Most of the time a 
single poll (say of 25 events) would be sufficient. In the rare case 
when this pool was exhausted, a second one could be allocated. If that 
one was used up, a third could be allocated...

> Any way, ISER seems to be needing a solution for this, and I think it is
> possible to come up with a different solution than vmalloc (maybe a few
> kmallocs) I will think about it and send a patch when I have one.

Ok. That would be great.