[openib-general] RE: some questions/comments on gen2 udapl code

[Davis, Arlin R]

Or,

We need to have these discussions on openib-general.

>-----Original Message-----
>From: Or Gerlitz [mailto:ogerlitz at voltaire.com]
>Sent: Thursday, August 04, 2005 6:55 AM
>To: Davis, Arlin R
>Cc: James Lentini
>Subject: RE: some questions/comments on gen2 udapl code
>
>> Arlin writes:
>> I take this back. With the current verbs event mechanisms, a
>> dedicated thread for uAT, uCM, and uCQ is the only way to process
>> multi-threaded application requests. Let's say an application opens
>> the device and then kicks off separate threads to connect and send to
>> many different endpoints. Each thread will be waiting on the same
>> event mechanisms. I see no good way to multiplex properly outside of
>> a dedicated processing thread.
>
>> If the verbs AT, CM, and CQ event mechanisms change then we can
>> re-visit.
>
>I am not sure there's is no good way - but i guess "not sure" is not
>enough, i need to send a
>patch that allows for it (ie work under multi-threaded app (eg
>udapltest) with each thread having
>diff conn/dto/cr evd)...

Patch would be great. I moved to a dedicated CQ thread approach after
struggling with the many multi-threaded issues with direct waits. See 
comments in dapls_ib_wait_object_wait() in dapl_ib_cq.c. 

>
>The thing is that the current implementation is not optimal in
>delivering CQ interrupts latency,
>since there's one "extra" context switch. Also the udapl provider

I agree that this is not optimal and we want the CQ_WAIT_OBJECT
mapped directly to a CQ fd to avoid the extra context switch. 
But with one CQ FD per device how do you optimally multiplex 
the user's CQ context across multiple waiters without a dedicated
processing thread?

It seems just as inefficient to have 4 waiters on different CQ's 
all waking up, processing an event that may or may not be theirs.
How do you deliver the event that was not yours? What if the 
event was directed to a thread that already woke and went back 
to sleep? Do you wake them all up again? This is why I went 
back to the dedicated CQ thread.  

We need some help from verbs to make this uDAPL CQ mapping work:

Either

 support multiple CQ FD's, one per CQ.

or

 modify call ibv_get_cq_event() and allow the user to supply the
 specific ibv_cq event to get, not just the first one on the event
queue.


>library opens 3 dedicated
>threads (uat/ucm/ucq) and for getting async event we will probably add
>uasync thread - which

I can work on rolling up the uat/ucm/uasync processing into one thread.

-arlin

>would be better if we can avoid doing (opening so many threads).
>
>Or.