[openib-general] Re: OpenIB Thread Usage

[Sean Hefty]

Roland Dreier wrote:

>     Sean> I thought about this approach, but wasn't sure about taking
>     Sean> a context switch.  I guess with QP redirection, this
>     Sean> wouldn't be an issue though.
> 
> I don't think there's a choice.  If the CM processes MADs from one queue
> and time wait expirations from another, it's not possible to prevent
> the MAD queue from getting arbitrarily far ahead of the time wait
> queue.  This results in QPs never being reaped and eventually the
> system runs out of memory.

I'm not understanding the issue here.  If connections are being made 
faster than QPs are leaving the time wait state, then the system will 
eventually run out of resources.  But this problem seems somewhat 
separate from the threading model used to establish connections, unless 
that thread is preventing other threads from executing.

If that's the case, is it be worth considering exposing the MAD work 
queue for use by not just the MAD layer, but also specific clients, such 
as the CM and SA client code?

I would think that as long as the code is structured around using work 
queues, we should be able to adjust the number of threads per work 
queue, along with the number of work queues in order to see what 
combination works best.

- Sean