[ofw] [RFC] [PATCH 1/2] work_queue: new abstraction for queuing work

[Fab Tillier]

>> Complib already has what you need: Quick pool.
>
> I don't know that it's desirable to allocate 1 work item per
> connection.  The HCA likely has a limited number of requests that it can
> service. Queuing, say, 3000 modify QP calls at once after a large MPI
> run can be worse on overall performance.  That many work items
> essentially blocks all system threads relative to how quickly a single
> HCA can service the modify requests. This was why I used a fix work item
> pool per WV provider.

Just think of how much simpler this all would be if the HCA driver supported async QP modify...  No work items needed.

In any case, as long as the work items each do a single modify QP, you should be fine queuing as many as you need - the thread pool can process items from the various priority queues as needed in between individual modify requests.  What's the difference between having 3000 requests queued up in your abstraction that keeps a work item queued per CPU until all are complete, or 3000 work items queued to the thread pool directly?  Either way, you'll have 3000 work items pass through the work item queue, you just delay queuing them in your implementation.

I would just allocate the work items dynamically for now, and then move the work item into the HCA driver by defining the async modify interface.  The HCA driver can then eventually be fixed to not need work items at all.

-Fab