[openib-general] uVerbs and multiple cq_event support?

[Arlin Davis]

Libor Michalek wrote:

>On Fri, Jul 15, 2005 at 01:26:11PM -0700, Arlin Davis wrote:
>  
>
>>Hi Roland,
>>
>>I have a question about uverbs cq_events. Currently, uverbs has one 
>>event (fd) handle that processes all CQ events. You mentioned earlier  
>>that you would possibly add support for more CQ events. Are you thinking 
>>one per CQ, or limiting to a small number like one per processor? If the 
>>thinking is NOT one per CQ then it may be somewhat restricitive to user 
>>applications that have multple connections with many CQs to process.
>>
>>For instance, with uDAPL there are user wait objects (CQ_WAIT_OBJECT) 
>>that can be mapped directly to these CQ events. This enables the user 
>>thread to wait and wake up directly from the kernel CQ event processing 
>>which is very efficicent. However, this can only be done if verbs 
>>provides an event mechanism per CQ. Otherwise, I have to process the 
>>events in a separate thread and in turn process the user's event 
>>independent of the CQ event.
>>
>>If the thinking is one per CQ then is fd's the way to go since we can 
>>have 1000's of CQs?
>>    
>>
>
>  An fd per CQ is definetly not the way to go for this reason. I don't
>think that the wait object would need to map to an individual fd for
>each CQ. The fd should become ready for read when there's a CQ event
>pending and a read of the fd will tell you which CQ it was that had
>the event. Only the wait object which represents that CQ needs to
>notify the consumer, the rest can go back to waiting or continue waiting
>depending how you implement the wait and locking around the fd. The
>point being is that you do not need another thread.
>  
>
This is actually how I implemented dapls_ib_wait_object_wait() in uDAPL 
(uCM and uAT version)
but I am not convinced it is any better then just using another thread. 
The method avoids another
thread but I now have a thundering herd all chasing the same event, 
sharing a mutex, and going
back to sleep if it was not yours. In addition, the user provided wait 
times have to be adjusted
for the next poll on the fd. Gets really ugly fast.

-arlin

>  I'm just looking at the ABI, so I'm not sure which function
>you should use to read the fd and get the info...
>  
>
>-Libor
>
>  
>